I have a Python3 script that can take optional command line parameters which the script reads via sys.argv. My script works fine if I run it from a terminal window, however any time I add it to the PA schedule tab with parameters, I get a bash 126 exit code in the log. According to the bash appendix this exit code means Permission problem or command is not an executable. So, it would appear that the scripts running from a terminal window don't run in the same context as scripts run from schedule, because my script runs fine from a bash terminal.
What I need is to figure out a way to get scripts to run with optional command line parameters via the schedule tab.
I figure I could write another script that calls the first script with the optional parameters, however it seems to me there should be another way to get around this where you can just put the parameters you wish to run the script with directly on the line you give to the schedule tab to run. Otherwise I'll have to write a script for each separate set of parameters I want to call with versus just an entry to schedule the script to run.
Thanks to #Lattyware for this solution. If you have this problem you can solve it by going to a terminal and adding the execute bit on your script. You can accomplish adding the execute permission using chmod such as:
chmod 700 foo.py
You may also want to verify your file permissions prior to any changes to ensure you don't add or remove something you didn't intend to.
You can read up on chmod here if you need to.
Related
I have a python script which executes the following command:
commands.getstatusoutput("./some_program -bg; sleep 10; kill -SIGUSR2 `pidof
some_program`; kill -9 `pidof some_program`")
In short, this calls a program (some_program) and puts it in the background, then, after 10s, it sends a signal to the program to prompt it to dump some data into a csv file, after which we kill the program.
When I run the python script manually, it dumps the output to the csv file as expected. When I run the program through a Jenkins build however, the csv file is not created.
Any ideas?
Sounds like a permission problem to me, are you running jenkins with a different user? I would chmod 777 the folder temporally only to see if thatś the issue
– gerosalesc
It turns out it was an ownership issue. – user2548144
I know how to set-up run configurations to pass parameters to a specific python script. There are several entry points, I don't want a run configuration for each one do I? What I want to do instead is launch a python script from a command line shell script and be able to attach the PyCharm debugger to the python script that is executed and have it stop at break points. I've tried to use a pre-launch condition of a utility python script that will sleep for 10 seconds so I can attempt to "attach to process" of the python script. That didn't work. I tried to import pdb and settrace to see if that would stop it for attaching to the process, but that looks to be command line debugging specific only. Any clues would be appreciated.
Thanks!
You can attach the debugger to a python process launched from terminal:
Use Menu Tools --> Attach to process then select python process to debug.
If you want to debug a file installed in site-packages you may need to open the file from its original location.
You can to pause the program manually from debugger and inspect the suspended Thread to find your source file.
I would like to create a simple Python program that will concurrently execute 2 independent scripts. For now, the two scripts just print a sequence of numbers but my intention is to use this program to concurrently run a few Twitter streaming programs in the future.
I suspect I need to use subprocess.Popen but I cannot quite get my head around what arguments I should put in there. There was a similar question on StackOverflow but the code provided there (pasted below) doesn't print anything. I will appreciate your help.
My files are:
thread1.py
thread2.py
import subprocess
subprocess.Popen(['screen', './thread1.py']))
subprocess.Popen(['screen', './thread2.py'])
Use supervisord
supervisord is process control system just for the purpose of running multiple command line scripts.
It features:
multiple controlled processes
autorestarting failed runs
log stdout and stderr output
starting scripts in order (using priority)
command line utility to view latest log output, stop, start, restart the processes
This solution works only on *nix based systems, it is not available on Windows.
As wanderlust mentioned, why do you want to do it this way and not via linux command line?
Otherwise, the solution you post is doing what it is meant to, i.e, you are doing this at the command line:
screen ./thread1.py
screen ./thread2.py
This will open a screen session and run the program and output within this screen session, such that you will not see the output on your terminal directly. To trouble shoot your output, just execute the scripts without the screen call:
import subprocess
subprocess.Popen(['./thread1.py'])
subprocess.Popen(['./thread2.py'])
Content of thread1.py:
#!/usr/bin/env python
def countToTen():
for i in range(10):
print i
countToTen()
Content of thread2.py:
#!/usr/bin/env python
def countToHundreds():
for i in range(10):
print i*100
countToHundreds()
Then don't forget to do this on the command line:
chmod u+x thread*.py
You can also just open several Command Prompt windows to run several Python programs at once - just run one in each of them:
In each Command Prompt window, go to the correct directory (such as C:/Python27) and then type 'python YourCodeNo1.py' in one Command Prompt window, 'python YourCodeNo2.py' in the next one ect. .
I'm currently running 3 codes at one time in this way, without slowing any of them down.
When I run my python script in the shell terminal, it works
sudo myscript.py --version=22 --base=252 --hosts="{'hostA':[1],'hostB':[22]}"
But when I run in Hudson and Jenkins, using Execute Shell step, somehow, the string --hosts="{'hostA':[1],'hostB':[22]}" is interpreted as
sudo myscript.py --version=22 --base=252 '--hosts="{'hostA':[1],'hostB':[22]}"'
How do we overcome this so that our script would run in Jenkins and Hudson ?
Thank you.
Sincerely
It looks like you're encountering a battle-of-the-quoted-strings type situation due to your use of quotes directly and the fact that Jenkins is shelling out from a generated temp shell script.
I find the best thing to do with Jenkins is to create a bash script that wraps the commands you want to run (and you can also have it do any other environment-related setup you may want to have it do, such as source a config bash script that sets up other env vars).
You can have it accept arguments that may vary from the command line, which can be passed to it from the Jenkins config. So any of the interpolation then happens within the script -- you're just passing strings. (In particular, in this case, you'll have the hosts arg be "{'hostA':[1],'hostB':[22]}", which will be passed to the shell script, and then interpolated, with the double quotes re-included.
So, to that end, say you have a jenkins_run.sh script that runs a command like this:
myscript.py --version=$VERSION --base=$BASE --hosts="$HOSTS"
Where the variables are passed in as arguments and assigned prior to that (you could directly use $0, $1 et al if you want.
I would also be cautious using sudo in conjunction with a Jenkins run, since that could end up prompting for I/O. I would instead recommend setting the permissions on the script such that the using under which Jenkins is running can simply execute the script.
Not sure if anyone has run into this, but I'll take suggestions for troubleshooting and/or alternative methods.
I have a Windows 2008 server on which I am running several scheduled tasks. One of those tasks is a python script that uses pscp to log into a linux box, checks for new files and if there is anything new, copies them down to a local directory on the C: drive. I've put some logging into the script at key points as well and I'm using logging.basicConfig(level=DEBUG).
I built the command using a variable, command = 'pscp -pw xxxx name#ip:/ c:\local_dir' and then I use subprocess.call(command) to execute the command.
Now here's the weird part. If I run the script manually from the command line, it works fine. New files are downloaded and processed. However, if the Task Scheduler runs the script, no new files are downloaded. The script is running under the same user, but yet yields different results.
According to the log files created by the script and on the linux box, the script successfully logs into the linux box. However, no files are downloaded despite there being new files. Again, when I run it via the command line, files are downloaded.
Any ideas? suggestions, alternative methods?
Thanks.
You can use the windows Task Scheduler, but make sure the "optional" field "Start In" is filled in.
In the Task Scheduler app, add an action that specifies your python file to run "doSomeWork" and fill in the Start in (optional) input with the directory that contains the file.. So for example if you have a python file in:
C:\pythonProject\doSomeWork.py
You would enter:
Program/Script: doSomeWork.py
Start in (optional): C:\pythonProject
I had the same issue when trying to open an MS Access database on a Linux VM. Running the script at the Windows 7 command prompt worked but running it in Task Scheduler didn't. With Task Scheduler it would find the database and verify it existed but wouldn't return the tables within it.
The solution was to have Task Scheduler run cmd as the Program/Script with the arguments /c python C:\path\to\script.py (under Add arguments (optional)).
I can't tell you why this works but it solved my problem.
I'm having a similar issue. In testing I found that any type of call with subprocess stops the python script when run in task scheduler but works fine when run on the command line.
import subprocess
print('Start')
test = subprocess.check_output(["dir"], shell=True)
print('First call finished')
When run on command line this outputs:
Start
First call finished
When run from task scheduler the output is:
Start
In order to get the output from task scheduler I run the python script from a batch file as follows:
python test.py >> log.txt
I run the script through the batch file both on command line and through task scheduler.
Brad's answer is right. Subprocess needs the shell context to work and the task manager can launch python without that. Another way to do it is to make a batch file that is launched by the task scheduler that calls python c:\path\to\script.py etc. The only difference to this is that if you run into a script that has a call to os.getcwd() you will always get the root where the script is but you get something else when you make the call to cmd from task scheduler.
Last edit - start
After experiments... If you put there full path to python program it works without highest privileges (as admin). Meaning task settings like this:
program: "C:\Program Files\Python37\python.exe"
arguments: "D:\folder\folder\python script.py"
I have no idea why, but it works even if script uses subprocess and multiple threads.
Last edit - end
What I did is I changed task settings: checked Run with highest privileges. And task started to work perfectly while running python [script path].
But keep in mind, that title contains "Administrator: " at the begining... always...
P.S. Thanks guys for pointing out that subprocess is a problem. It made me think of task settings.
I had similar problem when one script is running from Windows Task Scheduler, and another one doesn't.
Running cmd with python [script path] didn't work for me on Windows 8.1 Embedded x64. Not sure why. Probably because of necessity to have spaces in path and issue with quotes.
Hope my answer helps someone. ;)
Create a batch file add your python script in your batch file and then schedule that batch file .it will work .
Example : suppose your python script is in folder c:\abhishek\script\merun.py
first you have to go to directory by cd command .so your batch file would be like :
cd c:\abhishek\script
python merun.py
it work for me .
Just leaving this for posterity: A similar issue I faced was resolved by using the UNC (\10.x.xx.xx\Folder\xxx)path everywhere in my .bat and .py scripts instead of the letter assigned to the drive (\K:\Folder\xxx).
I had this issue before. I was able to run the task manually in Windows Task Scheduler, but not automatically. I remembered that there was a change in the time made by another user, maybe this change made the task scheduler to error out. I am not sure. Therefore, I created another task with a different name, for the same script, and the script worked automatically. Try to create a test task running the same script. Hopefully that works!
For Anaconda installation of python in windows below solution worked for me
create a batch file with below
"C:\Users\username\Anaconda3\condabin\activate" && python "script.py" &&
deactivate
Setup task to run this batch file