Develop Reference

python3 finds some programs in a folder, but not others

I have the following directory:
~/Library/Python/3.9/bin
Inside that folder, for example, appear these two programs:
Executing:
python3 -m submit50
gives...
usage: submit50 [-h] [--logout] [--log-level {debug,info,warning,error}] [-V]
slug
...
However, executing:
python3 -m youtube-dl
gives...
/Library/Developer/CommandLineTools/usr/bin/python3: No module named youtube-dl
What's happening here ?
Why isn't the program youtube-dl being located, and
Why is the path /Library/Developer/CommandLineTools/usr/bin/ appearing there ?
OS: macOS Ventura 13.1

Python and terminal: keep python environment after the file has been executed

I have to run a Python file from the Windows terminal (Windows PowerShell). I want that after the file has been executed (python foo.py), python keeps open with the variables defined in the file.
If it is unclear what I want, I want the same behavior as IDLE, after the file has been executed, you can write code in the IDLE command line and the variables defined in the file are stored in current session.
I need this for Windows now, but I might also need this for Ubuntu.

Apply the -i parameter as follows:
python -i foo.py
(Transcribed from python help):
python -h
usage: python [option] ... [-c cmd | -m mod | file | -] [arg] ...
Options (and corresponding environment variables):
…
-i : inspect interactively after running script; forces a prompt even
if stdin does not appear to be a terminal; also PYTHONINSPECT=x
…

Docker `/bin/sh: 1: [<executable>, : not found`

I want to run an executable installed with pip. Running the following image gives me /bin/sh: 1: [hbd,: not found:
FROM python:3.8.3-slim-buster
WORKDIR /data
COPY cookies.txt /data/cookies.txt
RUN python3.8 -m pip install humblebundle-downloader
CMD ["hbd", "download", "--cookie-file", "/data/cookies.txt", "--library-path", "/data" "--progress", "--update"]
I've tried CMD ["/usr/local/bin/python3.8", "-m", "hbd", "download",... and CMD python -m hbd download... and similar as well. Resulting in /bin/sh: 1: [/usr/local/bin/python3.8,: not found etc.
If I replace CMD ... with CMD which python3.8 && which hbd && find / -executable -type f -name hbd I get:
/usr/local/bin/python3.8
/usr/local/bin/hbd
/usr/local/bin/hbd
Which to me, tells me that the executables are there!?
How do I execute the, clearly present, executable?
I'm at my wits' end. Thank you!
Edit:
I tried with the 'non-slim' version of the base image, i.e FROM python:3.8.5-buster, no dice.

Hello I trying your Dockerfile it failed, I ran a shell inside the instead and tried hbd, the command was available just fine:
docker run -ti 7552b04ea25b sh
# hbd
usage: hbd [-h] {download} ...
hbd: error: the following arguments are required: action
I then tried without the quote, square brackets and commas, it worked:
CMD hbd download --cookie-file /data/cookies.txt --library-path /data --progress --update
I tried again with your CMD, it failed, I shortened it, it worked, with a few trial and errors I finally I saw that you forgot a comma between 2 arguments. You command shall be (notice the comma between 2 arguments has been added):
CMD ["hbd", "download", "--cookie-file", "/data/cookies.txt", "--library-path", "/data", "--progress", "--update"]
So next time, try you hypothesis and try to investigate yourself:
Hypothesis: The executable is not found ? Then test it by checking the command inside the container by running a shell inside it. Conclusion: path & executable are found.
Hypothesis: CMD syntax is wrong ? Let's try a different syntax. It works with CMD hbd. So yes obvious that was the CMD syntax... Let's try to debug it and craft it by dichotomy... Hey a comma is missing ;) Problem solved.

CMD ["hbd", "download", "--cookie-file", "/data/cookies.txt", "--library-path", "/data" "--progress", "--update"]
Should be
CMD ["hbd", "download", "--cookie-file", "/data/cookies.txt", "--library-path", "/data", "--progress", "--update"]
What I think is happening is that since CMD is not specified as a valid JSON array, it is interpreted as a normal command that begins with [ which is of course not valid.

The issue might be because of which the environment variables are not set or the path is not exported to the environment. Trying either of these, might help

python script doesn't execute when calling it from shell

I have a python 3 program in file foo the file has execution permissions, and the first line of the script is
#!/usr/bin/python3
When I run the file from python directly, i.e:
/usr/bin/python3 foo 3 boo
Everything runs perfectly well, but when I try to run the program by calling the file name I get:
foo 3 boo
foo: Command not found.
Even when specifying the relative or full path to the script I get the same response.
./foo 3 boo
./foo: Command not found.
/full/path/to/the/file/foo 3 boo
/full/path/to/the/file/foo: Command not found.
Some more info that was requested:
head -2 foo
#!/usr/bin/python3
which python3
/usr/bin/python3
Working from tcsh, where is the problem?
I'm working on a remote university computer through ssh, could it be the case that the sysadmins somehow prevent this?

You need to specify the absolute path of the script. Try
./foo 3 boo
while in the same directory as the script. Otherwise, the shell will only look in your PATH for something named foo, which doesn't exist.

You must give the path to the file
./foo
If you just call "foo", the shell will look in the environment variable $PATH the folder that contains the "foo" binary, and he will not find it ...

I'm using python3, just an example illustration..
Just a test script ..
$ cat test.py
#!/python/v3.6.1/bin/python3
import os
print(os.system("ls"))
$ which python3
/python/v3.6.1/bin/python3
Permission to make it executable.. i'm have doubt if the script foo is executable in your case..
$ chmod +x test.py
Test Run..
$ ./test.py
a.py Backup File_Write_Method NWK-old Python_aritsTest Python_Ftp Python_Mail Python_Primer test.py
argpass BASH Network_DeOps Python-3.6.3 Python_Dump Python_FTP Python_Panda readme_python tmp
awk.sh dnspython nslookup Python-3.6.3.tar.xz Python_excep Python_ldap Python_Parsers Regular_Expr tt.csv
0
It might be shell problem as i reproduced the tcsh shell & it fails to run..
$ tcsh
$ ./test
./test: Command not found.
but it runs , when i run it like below..
tcsh -c ./test.py

Assuming your script is executable and you have python3 in /usr/bin/python3. I think you are trying to run your script from a partition mounted with the noexec argument to check if it is the problem you should run:
mount | grep <partition> | grep noexec
If I'm right you can just mount the partition with the exec option or move your script elsewhere.

How can I check the syntax of Python script without executing it?

I used to use perl -c programfile to check the syntax of a Perl program and then exit without executing it. Is there an equivalent way to do this for a Python script?

You can check the syntax by compiling it:
python -m py_compile script.py

You can use these tools:
PyChecker
Pyflakes
Pylint

import sys
filename = sys.argv[1]
source = open(filename, 'r').read() + '\n'
compile(source, filename, 'exec')
Save this as checker.py and run python checker.py yourpyfile.py.

Here's another solution, using the ast module:
python -c "import ast; ast.parse(open('programfile').read())"
To do it cleanly from within a Python script:
import ast, traceback
filename = 'programfile'
with open(filename) as f:
source = f.read()
valid = True
try:
ast.parse(source)
except SyntaxError:
valid = False
traceback.print_exc() # Remove to silence any errros
print(valid)

Pyflakes does what you ask, it just checks the syntax. From the docs:
Pyflakes makes a simple promise: it will never complain about style, and it will try very, very hard to never emit false positives.
Pyflakes is also faster than Pylint or Pychecker. This is largely because Pyflakes only examines the syntax tree of each file individually.
To install and use:
$ pip install pyflakes
$ pyflakes yourPyFile.py

python -m compileall -q .
Will compile everything under current directory recursively, and print only errors.
$ python -m compileall --help
usage: compileall.py [-h] [-l] [-r RECURSION] [-f] [-q] [-b] [-d DESTDIR] [-x REGEXP] [-i FILE] [-j WORKERS] [--invalidation-mode {checked-hash,timestamp,unchecked-hash}] [FILE|DIR [FILE|DIR ...]]
Utilities to support installing Python libraries.
positional arguments:
FILE|DIR zero or more file and directory names to compile; if no arguments given, defaults to the equivalent of -l sys.path
optional arguments:
-h, --help show this help message and exit
-l don't recurse into subdirectories
-r RECURSION control the maximum recursion level. if `-l` and `-r` options are specified, then `-r` takes precedence.
-f force rebuild even if timestamps are up to date
-q output only error messages; -qq will suppress the error messages as well.
-b use legacy (pre-PEP3147) compiled file locations
-d DESTDIR directory to prepend to file paths for use in compile-time tracebacks and in runtime tracebacks in cases where the source file is unavailable
-x REGEXP skip files matching the regular expression; the regexp is searched for in the full path of each file considered for compilation
-i FILE add all the files and directories listed in FILE to the list considered for compilation; if "-", names are read from stdin
-j WORKERS, --workers WORKERS
Run compileall concurrently
--invalidation-mode {checked-hash,timestamp,unchecked-hash}
set .pyc invalidation mode; defaults to "checked-hash" if the SOURCE_DATE_EPOCH environment variable is set, and "timestamp" otherwise.
Exit value is 1 when syntax errors have been found.
Thanks C2H5OH.

Perhaps useful online checker PEP8 : http://pep8online.com/

Thanks to the above answers #Rosh Oxymoron. I improved the script to scan all files in a dir that are python files. So for us lazy folks just give it the directory and it will scan all the files in that directory that are python. you can specify any file ext. you like.
import sys
import glob, os
os.chdir(sys.argv[1])
for file in glob.glob("*.py"):
source = open(file, 'r').read() + '\n'
compile(source, file, 'exec')
Save this as checker.py and run python checker.py ~/YOURDirectoryTOCHECK

for some reason ( I am a py newbie ... ) the -m call did not work ...
so here is a bash wrapper func ...
# ---------------------------------------------------------
# check the python synax for all the *.py files under the
# <<product_version_dir/sfw/python
# ---------------------------------------------------------
doCheckPythonSyntax(){
doLog "DEBUG START doCheckPythonSyntax"
test -z "$sleep_interval" || sleep "$sleep_interval"
cd $product_version_dir/sfw/python
# python3 -m compileall "$product_version_dir/sfw/python"
# foreach *.py file ...
while read -r f ; do \
py_name_ext=$(basename $f)
py_name=${py_name_ext%.*}
doLog "python3 -c \"import $py_name\""
# doLog "python3 -m py_compile $f"
python3 -c "import $py_name"
# python3 -m py_compile "$f"
test $! -ne 0 && sleep 5
done < <(find "$product_version_dir/sfw/python" -type f -name "*.py")
doLog "DEBUG STOP doCheckPythonSyntax"
}
# eof func doCheckPythonSyntax