Running Talend jobs with Python - python

I'm trying to automate some stuff I would otherwise have to do manually, so I can run one python script instead of taking a whole bunch of steps. I want to find a way to run a Talend job from the python script.
How do I accomplish this? Is it even possible?

Oops! sorry.
From the Studio, build the job to get an autonomous job you can launch from command line.
Extract the files from the generated archive.
Search for folder "script/yourJobname".
Check the syntax from one of the .bat or .sh depending of which one you prefer.
Launch the jar file using subprocess.call (or other way to execute a jar file from Python).
Hope this helps.TRF

As soon as you can run a Python script from command line, you should be able to run it from Talend using tSystem component.

Related

How to run a python made .exe on a repeated schedule in Windows 10?

I have a GUI based.exe, which I made using Python auto-py-to-exe (one folder method), and I want the application to run up after every 10 minutes or so.
It is working when I click on it manually.
I have tried using task scheduler in windows to do that, but it is not exactly executing the application as I cannot see anything on screen. Any ideas how to do it?
Not answering directly the question but maybe still solving your problems. To run python scripts on schedule i usually write a .bat script that gets executed by task scheduler.
The bat script has to have the full length python path for the invocation and the full path of the .py file.
I think it's possible to use the same strategy to run exe files, so you could try using a .bat that targets your exe instead of targetting the exe directly in the task scheduler.

Executing a Powershell command from MATLAB

I'm trying to automate my workflow. So I'm generating a test list for my system in MATLAB. In the same script, I'm then waiting for a new log file which only gets generated when I run a python script in a certain directory through windows powershell.
That python script would take the test list generated and produce a new log file. My code looks for the date of modification of that file, if it has changed, it means a new log file has been produced. The code then extracts certain strings of characters and then plot the result.
My question is how can I invoke powershell from MATLAB to invoke that python script execution command in a specific directory?
Regards
If you really want to commit to this Powershell-in-the-middle approach, you can call it with system
system( 'powershell -command "Some Powershell command"' );
You can collect outputs from this if there are any.
Seems like it would be easier to call the Python script from Matlab though and cut out the call to Powershell, this could also be done with the system command (i.e. without also calling Powershell in between). Or you could invoke Python directly.

How to run or execute Talend jobs through python script?

I need to run a talend job outside of the talend studio. Is there any way to execute a talend job through python script ?
This is a common task and the following link gives detailed step by step procedure to create a .bat/.sh /jar.
https://community.talend.com/t5/Migration-Configuration-and/Exporting-a-Job-script-and-executing-it-outside-of-Talend-Studio/ta-p/21686
Once you create the job, then you can trigger it from python using subprocess.
import subprocess
subprocess.call(['(path)talendJob.jar'])
normally you can use any processmanagement tool to do that. For example
supervisor http://supervisord.org/. That's a processmanagement based
on python.

Rundeck :: Execute a python script

I am new to Rundeck, so I apologize if I ask a question that probably has an obvious answer I'm overlooking.
I've installed Rundeck on my Windows PC. I've got a couple of Python scripts that I want to execute via Rundeck.
The scripts run fine when I execute them manually.
I created a job in Rundeck, created a single step (script file option) to test the python script.
The job failed after six seconds. When I checked the log, it was because it was executing it line by line rather than letting python run it as an entire script.
How do I fix this?
You had to put:
#!/usr/bin/python
or similar, with location to your python binary, as 1st line. To indicate which interpreter to use for whole file.
okay, so I changed the step type to a command rather than script file and it worked.
I guess my understanding of what a script file is was off.

Problems running python script by windows task scheduler that does pscp

Not sure if anyone has run into this, but I'll take suggestions for troubleshooting and/or alternative methods.
I have a Windows 2008 server on which I am running several scheduled tasks. One of those tasks is a python script that uses pscp to log into a linux box, checks for new files and if there is anything new, copies them down to a local directory on the C: drive. I've put some logging into the script at key points as well and I'm using logging.basicConfig(level=DEBUG).
I built the command using a variable, command = 'pscp -pw xxxx name#ip:/ c:\local_dir' and then I use subprocess.call(command) to execute the command.
Now here's the weird part. If I run the script manually from the command line, it works fine. New files are downloaded and processed. However, if the Task Scheduler runs the script, no new files are downloaded. The script is running under the same user, but yet yields different results.
According to the log files created by the script and on the linux box, the script successfully logs into the linux box. However, no files are downloaded despite there being new files. Again, when I run it via the command line, files are downloaded.
Any ideas? suggestions, alternative methods?
Thanks.
You can use the windows Task Scheduler, but make sure the "optional" field "Start In" is filled in.
In the Task Scheduler app, add an action that specifies your python file to run "doSomeWork" and fill in the Start in (optional) input with the directory that contains the file.. So for example if you have a python file in:
C:\pythonProject\doSomeWork.py
You would enter:
Program/Script: doSomeWork.py
Start in (optional): C:\pythonProject
I had the same issue when trying to open an MS Access database on a Linux VM. Running the script at the Windows 7 command prompt worked but running it in Task Scheduler didn't. With Task Scheduler it would find the database and verify it existed but wouldn't return the tables within it.
The solution was to have Task Scheduler run cmd as the Program/Script with the arguments /c python C:\path\to\script.py (under Add arguments (optional)).
I can't tell you why this works but it solved my problem.
I'm having a similar issue. In testing I found that any type of call with subprocess stops the python script when run in task scheduler but works fine when run on the command line.
import subprocess
print('Start')
test = subprocess.check_output(["dir"], shell=True)
print('First call finished')
When run on command line this outputs:
Start
First call finished
When run from task scheduler the output is:
Start
In order to get the output from task scheduler I run the python script from a batch file as follows:
python test.py >> log.txt
I run the script through the batch file both on command line and through task scheduler.
Brad's answer is right. Subprocess needs the shell context to work and the task manager can launch python without that. Another way to do it is to make a batch file that is launched by the task scheduler that calls python c:\path\to\script.py etc. The only difference to this is that if you run into a script that has a call to os.getcwd() you will always get the root where the script is but you get something else when you make the call to cmd from task scheduler.
Last edit - start
After experiments... If you put there full path to python program it works without highest privileges (as admin). Meaning task settings like this:
program: "C:\Program Files\Python37\python.exe"
arguments: "D:\folder\folder\python script.py"
I have no idea why, but it works even if script uses subprocess and multiple threads.
Last edit - end
What I did is I changed task settings: checked Run with highest privileges. And task started to work perfectly while running python [script path].
But keep in mind, that title contains "Administrator: " at the begining... always...
P.S. Thanks guys for pointing out that subprocess is a problem. It made me think of task settings.
I had similar problem when one script is running from Windows Task Scheduler, and another one doesn't.
Running cmd with python [script path] didn't work for me on Windows 8.1 Embedded x64. Not sure why. Probably because of necessity to have spaces in path and issue with quotes.
Hope my answer helps someone. ;)
Create a batch file add your python script in your batch file and then schedule that batch file .it will work .
Example : suppose your python script is in folder c:\abhishek\script\merun.py
first you have to go to directory by cd command .so your batch file would be like :
cd c:\abhishek\script
python merun.py
it work for me .
Just leaving this for posterity: A similar issue I faced was resolved by using the UNC (\10.x.xx.xx\Folder\xxx)path everywhere in my .bat and .py scripts instead of the letter assigned to the drive (\K:\Folder\xxx).
I had this issue before. I was able to run the task manually in Windows Task Scheduler, but not automatically. I remembered that there was a change in the time made by another user, maybe this change made the task scheduler to error out. I am not sure. Therefore, I created another task with a different name, for the same script, and the script worked automatically. Try to create a test task running the same script. Hopefully that works!
For Anaconda installation of python in windows below solution worked for me
create a batch file with below
"C:\Users\username\Anaconda3\condabin\activate" && python "script.py" &&
deactivate
Setup task to run this batch file

Categories

Resources