I am attempting to write a python script that calls rsync to synchronize code between my vm and my local machine. I had done this successfully using subprocess.call to write from my local machine to my vm, but when I tried to write a command that would sync from my vm to my local machine, subprocess.call seems to erroneously duplicate text or modify the command in a way that doesn't make sense to me.
I've seen a number of posts about subprocess relating to whether a shell is expected / being used, but I tried running the command with both shell=True and shell=False` and saw the same weird command modification.
Here is what I am running:
command = ['rsync', '-avP', f'{user}#{host}:/home/{user}/Workspace/{remote_dir}', os.getcwd()]) # attempting to sync from a dir on my remote machine to my current directory on my local machine
print('command: ' + ' '.join(command))
subprocess.call(command)
Here is the output (censored):
command: rsync -avP benjieg#[hostname]:/home/benjieg/Workspace/maige[dirname] /Users/benjieg/Workspace
building file list ...
rsync: link_stat "/Users/benjieg/Workspace/benjieg#[hostname]:/home/benjieg/Workspace/[dirname]" failed: No such file or directory (2)
8383 files to consider
sent 181429 bytes received 20 bytes 362898.00 bytes/sec
total size is 217164673 speedup is 1196.84
rsync error: some files could not be transferred (code 23) at /AppleInternal/Library/BuildRoots/810eba08-405a-11ed-86e9-6af958a02716/Library/Caches/com.apple.xbs/Sources/rsync/rsync/main.c(996) [sender=2.6.9]
If I run the command directly in my terminal, it works just fine. The mysterious thing that i'm seeing is that subprocess appears to append some extra path before my user name in the command. You can see this in the 'link stat' line in the output. I'm guessing this is why it's unable to find the directory. But I really can't figure out why it would do that.
Further, I tried moving out of the 'Workspace' directory on my local machine to home, and even more weird behavior ensued. It seemed to begin looking into all of the files on my local machine, even prompting dropbox to ask me if iterm2 had permission to access it. Why would this happen? Why would whatever subprocess is doing cause a search of my whole local file system? I'm really mystified by this seemingly random behavior.
I'm on MacOS 13.0.1 (22A400) using python 3.10.4
I'm running the command
make config=default_config.mk task
with task being different tasks I can run. The entire program is pretty dated with very little support. The 'default_config.mk' file lets you change directorys, after that the programm is supposed to work.
It turns out the makefile runs the default-python. The Makefile basically runs a bunch of different .py-files.
I've already created an alias for python3 which works in the terminal, but not when I run the program. I've now created a new Makerun-file and changed every python to python3. Apart from that and adding '#!/bin/bash' to the first line (to make it executable?) it should be the same file.
Now, I keep getting the error
make: *** No rule to make target `task'. Stop.
Does anyone have an idea?
Thank you
Edit:
So I've managed to install Psycopg2 to Python 2.7, now it throws an error for the command:
python SciGRID.py --dbpwrd $(postgres_password)
Error:
SciGRID.py: error: --dbpwrd option requires an argument
The Programm is supposed to run on Python 2.7. I think the best way would be to change the Makefile, but that's when the error "no rule..." appears. If I use the default Makefile I basically keep running into errors as its pretty dated. Why is it not accepting the edited makefile?
I have a CodeDeploy which deploys application on Windows instances. I have a Python script which is running as part of ValidateService hooks. Below is the code I have in that script:
print("hello")
So, I have removed everything and just printing hello as part of this script. When this script is called by CodeDeploy I get below error:
My appspec.yml file:
...
ValidateService:
- location: scripts/verify_deployment.py
timeout: 900
I tried getting some help on Google but got nothing. Can someone please help me here.
Thanks
As Marcin already answered in a comment, I don't think you can simply run python scripts in CodeDeploy. At least not natively.
The error you see means that Windows does not know how to execute the script you have provided. AFAIK Windows can't run python natively (like most linux distros can).
I am not very accustomed to CodeDeploy, but given the example at https://github.com/aws-samples/aws-codedeploy-samples/tree/master/applications/SampleApp_Windows, I think you have to install python first.
After so much of investigations, I found my answer. The issue is little misleading, there is nothing to do with Code format or ENOEXEC. The issue was due to Python path. While executing my script, CodeDeploy was unable to find Python (Though I had already added python.exe in Environment variable path).
Also, I found that CodeDeploy is unable to execute .py file due to Python path issue. So, I created a PowerShell script and invoking Python script from there. Like below:
C:\Users\<username>\AppData\Local\Programs\Python\Python37-32\python.exe C:\Users\<username>\Documents\verify_deployment.py
It executed Python script successfully and gave me below output:
hello
I'm getting an interesting problem and I can't determine whether it's a problem with my code or the executable that I'm running. Basically I have a Python program that needs to call an external executable to process some data. If I call the executable via PowerShell or cmd, it works fine. However, if I attempt to run the executable via os.system() or subprocess.run(), I get the following error:
This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.
From my understanding of googling the issue, it would appear that this is some sort of C++-related issue, which is the language of the .exe that I'm running. I reinstalled the latest Visual C++ Redist and that did not seem to affect the problem. I've also tried to create a .bat and .ps1 script that runs the .exe. These both run fine via PowerShell and CMD, but raise the same error when run via os.system() and subprocess.run(). The error message is rather nondescript so I'm wondering if anyone knows anything about it and why os.system() etc. might be throwing it.
The relevant code is simply
os.system("GaussBin.exe gaussInput.txt gaussOutput.txt")
When the string is pasted into cmd, it runs perfectly. Additionally, if the parameters of the system() call are incorrect, the exe properly displays the usage function. It's only when I add the output.txt and the program is supposed to run fully that things start to break.
I've had some confusion about what directory os.system runs in. Should I be using .\ when calling the exe?
The .exe file is provided, not built by me.
So this is an unusual one, and perhaps I am simply missing the obvious, but I have the following python code that creates a powershell script and runs it.
# Create the PowerShell file
f = open("getKey.ps1", "w")
f.write('$c = Get-BitlockerVolume -MountPoint C:\n')
f.write('$c.KeyProtector[1].RecoveryPassword | Out-File C:\\Temp\\recovery.key\n')
# Invoke Script
startPS = subprocess.Popen([r'C:\Windows\system32\WindowsPowerShell\v1.0\powershell.exe',
'-ExecutionPolicy', 'Unrestricted', './getKey.ps1'], cwd=os.getcwd())
result = startPS.wait()
When this is run, it gives me the following error:
The term 'Get-BitlockerVolume' is not recognized as the name of a cmdlet, function, script file, or operable program.
However, if I then go and manually run the generated script, it works perfectly. To add to the oddity, if I run the same command exactly as above ie:
C:\Windows\system32\WindowsPowerShell\v1.0\powershell.exe -ExecutionPolicy Unrestricted ./getKey.ps1
it also works exactly as expected.
Clearly, the above error is a powershell error, so it is successfully running the script. It almost seems like powershell somehow knows that this is being run from python and has some restricted library of commands when a script is run from a particular source. I grant that that idea makes no real sense, but it's certainly how things appear.
I don't think this is a permissions issue, because when you run the same command from an unelevated powershell prompt, you get an Access is denied type error, rather than a command doesn't exist kind of error.
Anyway, any help would be greatly appreciated!
Edits
Edit: New evidence to help figure this out:
It's definitely an issue of cmdlets being loaded properly. If I programmatically run a script to dump the list of all available commands to a text file, it is only about 2/3's as big as if I do so through a powershell prompt directly
I bet Python is running as a 32-bit process on 64-bit Windows. In this case, you'll end up running 32-bit PowerShell, which in practice is a Bad Thing since many PowerShell modules depend on native binaries that may not have 32-bit equivalents. I hit this with IIS Manager commandlets--the commandlets themselves are registered in 32-bit PowerShell, but the underlying COM objects they rely on are not.
If you need to run 64-bit PowerShell from a 32-bit process, specify the path as %SystemRoot%\SysNative\WindowsPowerShell\v1.0\PowerShell.exe instead of System32.
System32 is actually virtualized for 32-bit processes and refers to the 32-bit binaries in %SystemRoot%\SysWow64. This is why your paths (and PSMODULEPATH) will look the same, but aren't. (SysNative is also a virtualized path that only exists in virtualized 32-bit processes.)
Adding to what #jbsmith said in the comment, also check to make sure that the environment variable that PowerShell relies on to know where it's modules are is populated correctly when python starts the process.
%PSMODULEPATH% is the environment variable in question, and it works the same way the %PATH% variable does, multiple directories separated by ;. Based on what you say your observed behavior is, it seems that you are using PowerShell 3.0, and cmdlet autoloading is in effect.
The solution here: Run a powershell script from python that uses Web-Administration module got me the cmdlet I needed, however there are still missing cmdlets even when using this method. I'm still at a loss as to why some are loaded and others are not, but for the time being, my script does what I need it to and I can't spend any more time to figure it out.
For reference here is the code that worked for me
startPS = subprocess.Popen([r'C:\Windows\sysnative\cmd.exe', '/c', 'powershell',
'-ExecutionPolicy', 'Unrestricted', './getKey.ps1'], cwd=os.getcwd())
I had the same issue, and it was simply that the BitLocker feature was not installed, hence the module wasn't present.
I fixed it by installing the Bitlocker feature:
Windows Server:
Install-WindowsFeature BitLocker -IncludeAllSubFeature -IncludeManagementTools -Restart
Windows Desktop:
Enable-WindowsOptionalFeature -Online -FeatureName BitLocker -All