Develop Reference

Wyh Is a subprocess call to python not using the python interpreter the calling script was executed in?

I am trying to run python code on a build server. In order to keep the agent clean, I'm creating a virutal environment which can be deleted after the task. The python script calls python via subprocess. The Questions are:
why does the call to subprocess not use the same python virtual env the actual script was called in?
How can this be achieved?
Miminal example:
tmp.py:
from subprocess import check_output
import sys
# python interpreter used to call this script
print(sys.executable)
# check which python interpreter is used when calling subprocess
print(check_output(f'python -c "import sys\nprint(sys.executable)').decode())
run.bat:
#echo off
python -m venv .\test_venv
call .\test_venv\Scripts\activate.bat
python tmp.py
output, where the second line is the default python installation on my computer:
λ run.bat
D:\tmp\pytest\test_venv\Scripts\python.exe
D:\tools\python\python.exe
desired output:
λ run.bat
D:\tmp\pytest\test_venv\Scripts\python.exe
D:\tmp\pytest\test_venv\Scripts\python.exe
I am on 64 bit Windows 10.

The subprocess you create uses the operating system's general PATH traversal to find and run the commands you specify, and doesn't know anything about the parent process.
You already know the value of sys.executable; if that's specifically what you want to run, say so:
print(check_output([sys.executable, "-c", "import sys\nprint(sys.executable)"]), text=True)
(This also avoids the shell, which was providing no value at all. Without an explicit shell=True, your code would only work on Windows.)
(Conversely, on any sane platform, the environment, including the virtual environment, would be inherited by child processes.)
However, Python calling Python is almost always an antipattern. Instead, you want to refactor the code so you can import it and run it in the same process.

Is there a way to execute .py scripts from path in windows?

I would like to use a python script anywhere within command prompt. This is possible in unix, but I couldn't find anything for windows. Do I have to convert it to .exe?
Edit: not sure why this is being downvoted, maybe it's a silly question but I can't find any similar threads here and I can't be the first person to want to execute .py scripts from their path...
Edit 2: I think my wording was unclear. I would like to know a method to execute python scripts in Windows without needing to specify python path/to/script.py every time. Here is a solution on Linux where the shebang statement invokes the python interpreter, and the script in question can be easily placed in bin: How do I install a script to run anywhere from the command line? . Does there exist a solution like this for Windows?

Here's a solution for running myScript.py:
add to the myScript.py file a first line #!python (or #!python3 if you want to use Python 3)
For instance:
#!python
import sys
sys.stdout.write("hello from Python %s\n" % (sys.version,))
change the "opens with" property of myScript.py to py.exe (to find where it is use where py-- I have it in C:\Windows\py.exe)
put the script myScript.py somewhere in your Windows path
and now you should be able to type myScript.py anywhere in a command prompt and run the Python script with your chosen Python version.
See: https://docs.python.org/3.6/using/windows.html#from-the-command-line

R: system() cannot use bash function defined in .bashrc

Question
My default Python is 2.7, but I have a script that requires Python 3.4. I am trying to create a function in R that will:
Switch to Python 3.4
Run this script
Switch back to Python 2.7
Import results into R
To switch between Python versions, I use my cluster's "dotkit" system, like this:
use Python-2.7
use Python-3.4
"use" is a bash function that is imported in my .bashrc file. It sets all of my path variables (PATH, LIBRARY_PATH, LD_LIBRARY_PATH, CPATH, C_INCLUDE_PATH, etc). The problem is that when I try to call this function in R, I get the following error:
system('use Python-3.4')
sh: use: command not found
It seems like this is a problem with my PATH. I am using the correct shell:
system('echo $SHELL')
/bin/bash
My $PATH variable also looks good. However, when I create a script that essentially does the same thing:
load_py34.sh:
#!/bin/bash
source ~/.bashrc
use Python-3.4
and call this script through R, then it actually runs, but for some reason, it doesn't change my python version within R. (I have verified that this script works from the command line.)
> R
> system('python --version')
Python 2.7.1
> system('sh load_py34.sh')
Prepending: R-3.4 (ok)
> system('python --version')
Python 2.7.1
So I'm a little confused, but if anyone can help, I would really appreciate it.
Suggested fixes
When I combine them into a single command, I still have the same problem:
> system("sh load_py34.sh; python --version")
Prepending: Python-3.4 (already loaded)
Python 2.7.1
When I try calling bash directly, I still have a problem with the PATH:
> system("bash -c 'use Python-3.4; python --version'")
bash: use: command not found
Python 2.7.1

.bashrc is only loaded for interactive bash sessions.
"use" is a bash function that is imported in my .bashrc file. It sets
all of my path variables.
If set via export, the environment of the calling process will not be altered.
export [-fn] [name[=word]] ... The supplied names are marked for automatic export to the environment of subsequently executed commands. (https://man7.org/linux/man-pages/man1/bash.1.html)
Child processes do not normally have access to the parent process' environment. (This poses a problem because system() creates a sub-process.)
The source and . built-ins execute the commands in the current shell environment, hence why your script works.
Other commands (executables, non-shell-builtins) are executed by the fork-and-exec mechanism, whereby the executing shell process forks, creating a child process with an identical environment and state. This new child process is the process in which the command is executed. Changes to the environment of that process are not replicated to the parent's environment.
This means that you will not be able to rely on system('...') to modify the environment of the R process, or that of processes spawned by subsequent system() invocations.
In a single invocation to system(), you can construct a command-line that changes the environment of the spawned shell like so:
bash -c 'source ~/.bashrc; use Python-3.4; python --version'
Mind you, ~/.bashrc is not really the best place to put this functionality (might be subjective).

When you call system() it uses /bin/sh, not /bin/bash. sh doesn't read your .bashrc file when it starts up, so it does not know any of the functions you've defined there.
To use the function from your .bashrc, you must get bash to run it instead:
system("bash -c 'use Python-3.4; python --version'")
Edit: placement of closing single quote.

Canopy get list of files in the shell

I'm using canopy python. The shell has commands "pwd" and "cd".
It even does autocomplete.
Is there a command - not something I have to write and keep track of - a built-in command that just lists the files in the directory.
I've tried ls, ls(), dir, dir(), directory, directory(), files, files(), filelist, filelist().
I've searched their documentation for listing files. It shows how to get a list of files in a python program - that's great. I do that all the time. I don't want to do that every time I want to list the contents of the current directory.
Is there a built-in shell command for this?

You can try
import os
fileList = os.listdir(directory_path)
so that you have the list of files in the python program to manipulate.
If you want the list of files in the current working directory in the way that ls would give you you can do.
fileList = os.listdir(os.getcwd())
If you really want a short command to type over and over you can write
ls = lambda : os.listdir(os.getcwd())
and then it will run this command when you type
ls()

You can use the python built in module, os, to query a directory using the system method. Please see below.
import os
os.system("dir")
Update
Please keep in mind that my solution is OS specific (i.e. ls vs dir) so Jon's solution is preferable.

The premise of the question seems inconsistent. If you can do the ipython pwd or cd standard magic commands, then you should also be able to access the ipython standard alias ls. If you cannot, then it almost certainly because you've overwritten / shadowed the meaning of the ls alias.
It's crucial to distinguish between (1) a plain Python shell (opened by typing python at a Canopy command prompt), which does not provide pwd nor cd nor 'lscommands; and (2) *any* IPython shell, whether the Python panel in the Canopy GUI or anipython` terminal in the Canopy Command Prompt, which does provide those, and many other Ipython "magic" commands.
(For more about IPython magic commands, see http://ipython.readthedocs.io/en/stable/interactive/magics.html; these are the docs for IPython version 6, but they are mostly the same for IPython version 5 which Canopy uses in order to support both Python 2.7 and Python 3.5+.)
The two previous answers are always correct in that they will work in any Python program or shell (depending on OS). That is because they only use the Python standard library. However they are not analogous to the cd and pwd commands that you mention, which are specific to the IPython shell (not to programs running in an IPython shell).
IMO, there is very seldom a good reason for opening a plain Python shell. If you simply want to run a script file from a Command Prompt, then by all means python myscript.py. But if you want an interactive shell, IPython provides so much extra capability that it has been for many years the de facto standard for Python shells (which is why it is used for the Canopy GUI's Python shell).

Got a note back from Canopy Support).
There is an environment variable, COMSPEC that was set to:
C:\windows\system32\cmd.exe;
That semicolon at the end is wrong. This is a single directory and not a list of directories. I removed that semicolon and suddenly the 'ls' command works as I remember!

Windows 7 - pydoc from cmd

Okay, I'm having one of those moments that makes me question my ability to use a computer. This is not the sort of question I imagined asking as my first SO post, but here goes.
Started on Zed's new "Learn Python the Hard Way" since I've been looking to get back into programming after a 10 year hiatus and python was always what I wanted. This book has really spoken to me. That being said, I'm having a serious issue with pydoc from the command. I've got all the directories in c:/python26 in my system path and I can execute pydoc from the command line just fine regardless of pwd - but it accepts no arguments. Doesn't matter what I type, I just get the standard pydoc output telling me the acceptable arguments.
Any ideas? For what it's worth, I installed ActivePython as per Zed's suggestion.
C:\Users\Chevee>pydoc file
pydoc - the Python documentation tool
pydoc.py <name> ...
Show text documentation on something. <name> may be the name of a
Python keyword, topic, function, module, or package, or a dotted
reference to a class or function within a module or module in a
package. If <name> contains a '\', it is used as the path to a
Python source file to document. If name is 'keywords', 'topics',
or 'modules', a listing of these things is displayed.
pydoc.py -k <keyword>
Search for a keyword in the synopsis lines of all available modules.
pydoc.py -p <port>
Start an HTTP server on the given port on the local machine.
pydoc.py -g
Pop up a graphical interface for finding and serving documentation.
pydoc.py -w <name> ...
Write out the HTML documentation for a module to a file in the current
directory. If <name> contains a '\', it is treated as a filename; if
it names a directory, documentation is written for all the contents.
C:\Users\Chevee>
EDIT: New information, pydoc works just fine in PowerShell. As a linux user, I have no idea why I'm trying to use cmd anyways--but I'd still love to figure out what's up with pydoc and cmd.
EDIT 2: More new information. In cmd...
c:\>python c:/python26/lib/pydoc.py file
...works just fine. Everything works just fine with just pydoc in PowerShell without me worrying about pwd, or extensions or paths.

In Windows Powershell use: python -m pydoc
Examples:
python -m pydoc open
python -m pydoc raw_input
python -m pydoc argv

When you type the name of a file at the windows command prompt, cmd can check the windows registry for the default file association, and use that program to open it. So if the Inkscape installer associated .py files with its own version of python, cmd might preferentially run that and ignore the PATH entirely. See this question.

Based on your second edit, you may have more than one copy of pydoc.py in your path, with the 'wrong' one first such that when it starts up it doesn't have the correct environment in which to execute.

python -m pydoc -k/p/g/w <name>

Syntax for pydoc on windows:
alt1:
C:\path\PythonXX\python.exe C:\path\PythonXX\Lib\pydoc.py -k/p/g/w X:\path\file_to_doc.py
alt2:
python -m pydoc -k/p/g/w X:\path\file_to_doc.py
Of which the latter is the one to prefer, duh. However it requires your windows installation to have registered python to the environment variable "Path".
Setup windows environment variables:
Look at this site for a guide on where to find them. The one you'll be looking for is "Path". If you select Path and click Edit you will see a long row of paths pointing to different folders. The Path's you see here is what allows you to basically reach a veriety of programs in the command line by just entering the name of the program, instead of the whole path to it. So what you want to do here is to locate your Python installation and copy its full path like this: X:\subfolders\PythonXX\ Then you add it to the very end of the long row of Path's like this:
X:\earlier\path\to\something;X:\subfolders\PythonXX\
Notice the ";" that seperates the different paths, make sure not to forget it. When done, click to confirm/Ok, then you would need to restart any cmd.exe that's already open.
The pydoc.py
The thing is that pydoc is a module of the standard python lib, and it's also powered by python. The windows environment, of what I understand, requires you to specify with which program you want to run a certain file with. So to run the pydoc.py-file we would use:
Open file in windows cmd.exe:
X:\subfolders\Python27\python.exe X:\subfolders\Python27\Lib\pydoc.py
pydoc.py's arguments:
pydoc.py comes with a veriety of command line-based features that allows you to enter certain arguments:
-k/p/g/w of which will trigger different behaviours of the pydoc.py-program.
Send arguments to a program through command line:
The syntax to enter these arguments is of what I know always after the x:\pathtofile\filename.suffix, seperated by a simple space. Which gives us the final:
alt1:
X:\subfolders\Python27\python.exe X:\subfolders\Python27\Lib\pydoc.py -w X:\path\file_to_doc.py
alt2 (with python registered to path):
python -m pydoc -w X:\path\file_to_doc.py
The "w"-option will give you a HTML-documentation for the file you want to run documentation on. Notice that pydoc.py will (according to my tests) create the documentation-file in the current working directory. Meaning that you will need to place yourself in a folder of choice before you actually run the command.
The function of -m
Of what I can find, the -m seem to handle registry entries, atleast in the msiexec.exe. I guess it might be used for programs in general this way. So my speculative idea of it is that if "-m" is applied, the pursuing arguments paths will be rewritten so that the .exe-file will be used as a path-reference. But as said, rather speculative.
-m Rewrites all required computer-specific registry entries. (in msiexec.exe) According to Microsoft