I have written a python code, then ran it through SSIS in the Execute Process Task, because I want to schedule it with table refresh dependency(which is not possible through windows scheduler or Alteryx).
The code reads and writes to SQL management studio to translate via Google API (pip install googletrans==4.0.0-rc1), with no output or input through SSIS. the input and output are done through sqlacademy in python.
The codes runs perfectly fine in Jupyter notebook, and I saved it to be .py file, but I face a lot of error in the Execute Process Task in SSIS.
Here are a couple of the solutions I tried:
in EXCUTABLE (in command prompt written where python copied .exe path)
in Arguments C:......\file.py
in WorkingDirectory C:......\ (only removing the file name while keeping the backslash
no white spaces a solution from stackflow)
Error:
"C:\Users\me\AppData\Local\Programs\Python\Python311\python.exe" "UC2.py" at "C:\Users\me\Downloads\folder\test", The process exit code was "1" while the expected was "0".
in EXCUTABLE (in command prompt written where python copied .exe path)
in Arguments 'file.py' (file name in quataion a solution from stackflow)
in WorkingDirectory C:......\ (only removing the file name while keeping the backslash
no white spaces a solution from stackflow)
Error:
The process exit code was "2" while the expected was "0"
move the .exe to a folder were no other files exists.
The process exit code was "-1073741515" while the expected was "0"
I also tried opening a new project.
then I tried right click change success value no error when running in SSIS, but the following error pops up:
enter image description here
everything even the sql server is installed on the remote desktop.
HelloWorld.err.py
A trivial python script that will write a string to standard out, another to standard error and the exit with return code 1.
import sys
print("Hello world")
print("Egads, something bad happened!", file=sys.stderr)
exit(1)
SSIS package
A sample SSIS package would look something like the following. An Execute Process Task has a successor of a Script Task.
Double click the precedent constraint (the line between the two) and change that from the default of Success to Completion
EPT Python
An Execute Package Task that is configured as shown
Executable: C:\Program Files\Python39\python.exe
Arguments: C:\ssisdata\SO_75361110.err.py
StandardOutputVariable: User::StdOut
StandardErrorVariabele: User::StdErr
SCR Echo Back
This is a generic Script Task that I use and abuse to dump the values of SSIS Variables to the output log/results tab. It's easy, lightweight and let's you print values without popups or any hard work.
All you need to do is add the Variable(s) in the readonly/readWrite collection. In this case, I selected User::StdOut and User::StdErr
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
namespace ST_ede4b38a087c4724a71149ffc5ed6be9
{
[Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
{
public void Main()
{
bool fireAgain = false;
foreach (Variable item in Dts.Variables)
{
Dts.Events.FireInformation(0, "SCR Echo Back", string.Format("{0}->{1}", item.QualifiedName, item.Value), "", 0, ref fireAgain);
}
Dts.TaskResult = (int)ScriptResults.Success;
}
enum ScriptResults
{
Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
};
}
}
Results
When my package runs, as "expected" the SuccessValue from my run will be 1 given that I use exit(1) from the python script.
Let's see what the Output tab looks like
SSIS package "C:\Users\bfellows\source\repos\SO_Trash\SO_Trash\SO_75361110.dtsx" starting.
Error: 0xC0029151 at EPT Python the thing, Execute Process Task: In Executing "C:\Program Files\Python39\python.exe" "C:\ssisdata\SO_75361110.err.py" at "", The process exit code was "1" while the expected was "0".
Task failed: EPT Python the thing
Warning: 0x80019002 at SEQC Do the thing: SSIS Warning Code DTS_W_MAXIMUMERRORCOUNTREACHED. The Execution method succeeded, but the number of errors raised (1) reached the maximum allowed (1); resulting in failure. This occurs when the number of errors reaches the number specified in MaximumErrorCount. Change the MaximumErrorCount or fix the errors.
Information: 0x0 at SCR Echo Back, SCR Echo Back: User::StdErr->Egads, something bad happened!
Information: 0x0 at SCR Echo Back, SCR Echo Back: User::StdOut->Hello world
Warning: 0x80019002 at SO_75361110: SSIS Warning Code DTS_W_MAXIMUMERRORCOUNTREACHED. The Execution method succeeded, but the number of errors raised (1) reached the maximum allowed (1); resulting in failure. This occurs when the number of errors reaches the number specified in MaximumErrorCount. Change the MaximumErrorCount or fix the errors.
SSIS package "C:\Users\bfellows\source\repos\SO_Trash\SO_Trash\SO_75361110.dtsx" finished: Failure.
The program '[68044] DtsDebugHost.exe: DTS' has exited with code 0 (0x0).
Of note, are these two lines
Information: 0x0 at SCR Echo Back, SCR Echo Back: User::StdErr->Egads, something bad happened!
Information: 0x0 at SCR Echo Back, SCR Echo Back: User::StdOut->Hello world
That's our Script Task echoing back the value of our Standard Out and Standard Error text streams. My suspicion is that given a "normal" run is 10 minutes and the runs from SSIS are measured in seconds, the process execution is throwing errors to the caller but nobody is listening. This will add a listener and give you better information to diagnose your root cause.
Related
I want to Schedule a python script.
I need help in ACTIONS tab: Here I have provided the below
I'm unable to run the task.
Please assist.
On a Windows 10 system with standard Anaconda installation, I ran the
following action, leaving all the other screens defaulted, except for
selecting trigger at a certain time for a one-time run:
Program/script: C:\ProgramData\Anaconda3\python.exe
Add arguments: "try csv 2.py" (double quotes since spaces in script name)
Start in: C:...\folder (in your case: C:\helloo)
On first try, I ran this task with the arguments without double quotes; this resulted in Last Run Result (0x2). I think this is the return code from execution, with 0 being success.
I am using a batch file to access my portable VLC executable to convert an mp4 to an mp3:
set arg1=%1 REM -> arg1={my_mp4_full_path}
set arg2=%2 REM -> arg2={my_mp3_full_path}
echo %arg1%
echo %arg2%
REM batch file is in the same directory as "VLCPlayer" folder
"%~dp0\VLCPlayer\VLCPortable.exe" -I dummy %arg1% --sout=#transcode{acodec=mp3,ab=128,vcodec=dummy}:std{access="file",mux="raw",dst=%arg2%} vlc://quit
When I run this script the first time, vlc crashes and I get an unplayable mp3 file, however when I run the script again the script works and I get a playable mp3. Is there a way to remedy this, or make it consistent? I don't see why running it twice would yield different outcomes.
No I don't have ffmpeg on my computer it is unrecognizable internal or external command.
Note that I face the same problem when using powershell to perform the same task, when I import my function from a .psm1 script:
function ConvertToMp3(
[switch] $inputObject,
[string] $vlc = '{PAth_TO_PORTABLE_VLC}\VLCPortable.exe')
{
PROCESS {
$codec = 'mp3';
$oldFile = $_;
$newFile = $oldFile.FullName.Replace($oldFile.Extension, ".$codec").Replace("'","");
&"$vlc" -I dummy "$oldFile" ":sout=#transcode{acodec=$codec,
vcodec=dummy}:standard{access=file,mux=raw,dst=`'$newFile`'}" vlc://quit | out-null;
# delete the original file
Remove-Item $oldFile;
}
}
I get the same random output that sometimes works, sometimes crashes.
Update:
I feel like I should add more info of how I use the batch file:
I have a python script Convert.py and I call my batch file inside using os.system():
mp4_to_convert = arguments.file
full_path_mp4 = os.path.join(outdir,mp4_to_convert)
mp3_to_convert_to = mp4_to_convert.replace(".mp4",".mp3")
full_path_mp3 = os.path.join(outdir,mp3_to_convert_to)
command_string = """Convert_Script.bat \"{}\" \"{}\"""".format(full_path_mp4, full_path_mp3)
os.system(command_string)
This is the documentation of os.system():
os.system(command)
Execute the command (a string) in a subshell. This
is implemented by calling the Standard C function system(), and has
the same limitations. Changes to sys.stdin, etc. are not reflected in
the environment of the executed command. If command generates any
output, it will be sent to the interpreter standard output stream.
On Unix, the return value is the exit status of the process encoded in
the format specified for wait(). Note that POSIX does not specify the
meaning of the return value of the C system() function, so the return
value of the Python function is system-dependent.
On Windows, the return value is that returned by the system shell
after running command. The shell is given by the Windows environment
variable COMSPEC: it is usually cmd.exe, which returns the exit status
of the command run; on systems using a non-native shell, consult your
shell documentation.
Any pointers or suggestions would be helpful, thank you in advance for your help.
There isn't a particular point in the script where it hangs (I've seen it getting stuck at random points in the script), so checking the logs didn't yield much insight. It doesn't even throw an exception or an error. It just keeps running while stuck.
I'm basically calling this python script from a powershell file (which later gets called by Task scheduler).
$python = "C:\Python34\python.exe"
$python_path = "C:\Source\main.py"
cd (split-path $python_path)
while($true)
{
& $python $python_path
}
Is there something I need to do to make sure it doesn't get stuck?
You Will have to do your path strings like this path ="c:\\test\\file"
Because it ignorer the first backslash
import os
import subprocess
import sys
import re
## fname_ext=sys.argv[1]
fname_ext=r"C:\mine\.cs\test.cs"
exe=os.path.splitext(fname_ext)[0]+".exe" # Executable
fdir=os.path.split(fname_ext)[0]
fcontent=open(fname_ext).read()
p_using=re.compile("\s*using\s+((\w+[.]*)+)")
p_namespace=re.compile("\s*namespace\s+(\w+)")
usings=p_using.findall(fcontent)
usings=[x[0] for x in usings]
references=[]
for i in os.listdir(fdir):
path=fdir+"\\"+i
try:
if os.path.isdir(path) or (not path.endswith('cs')):continue
with open(path) as fp:
content=fp.read()
namespaces=p_namespace.findall(content)
for n in namespaces:
if n in usings and 'System' not in n:
references+=[path]
except:
pass
command="csc /nologo "+" ".join(references)+" "+fname_ext
## command=" ".join(references)
#~ ---------------------------------------------------------
# Build:
option=1
if option==0:
# using os.system
print ">>",command
if os.system(command)==0:
os.system(exe)
else:
#~ Using subprocess module
## print type(references)
command=['csc']
## print command,references
command.extend(["/nologo","/out:"+exe])
command.extend(references)
command.append(fname_ext)
## print command
if subprocess.call(command,shell=True)==0:
## print "running %s"%exe
subprocess.call([exe],shell=True)
else:
pass
## print "Failed to run"
#~ ---------------------------------------------------------
I have this code above that is supposed to run a Csharp program from SciTE. It searches
every .cs file in the directory and finds the file with the namespace that the current
file has included. The command to run the file in SciTE is:
command.go.*.cs=python C:\mine\.py\csc.py $(FilePath)
command.go.subsystem.*.cs=0
That program logic part is okay.
The issue is that when hit F5 with sample Csharp code like this:
using System;
using System.Collections;
using MyNamespace;
class Test{
public static void Main(String[] args){
MyObject inst=new MyObject();
MyObject.self_destruct(inst);
}
}
it runs ok. But when I uncomment the second fname_ext and comment the first one
and run the csc.py file, a window opens and keeps running, printing command(this happens
using the os.system option). When you use the subprocess.call option, the same thing
happens but this time only when shell=True. It ran for only 15 seconds and there were 800+
cmd.exe and python.exe processes.I had to wait almost 5 minutes after killing cmd.exe
for the mouse to start responding and 2 minutes more for desktop peek to work.
When shell=False, it runs ok, the same way as when you hit the F5 key from the file.
What is happening here?
What is shell=True doing that makes it behave that way?
The problem is that your sys.argv looks something like this:
['python', r'C:\mine\.py\csc.py', 'whatever.cs']
So, with the fname_ext line uncommented, you set fname_ext to r'C:\mine\.py\csc.py'. Which means your script ends up just running itself—which again runs itself, etc., as fast as possible until your system chokes.
The reason it doesn't happen with shell=False is that you can't actually exec a Python script. Ultimately you end up calling CreateProcess with your script, which tries to interpret it as a .exe file, fails, and returns an error. But with shell=True, you pass your script to cmd.exe to run as a program, and it does the same thing an interactive prompt or Explorer would do: finds the right mapping to execute .py files and uses it. (And os.system does effectively the same thing as shell=True, but with a couple extra layers tossed in for good measure.)
Okay, I'll take a stab at this. If I understand the situation, this script is called csc.py and you want to call the csc c# compiler. When you run csc /nologo (etc...) through cmd.exe, it starts looking for something called 'csc' with a known extension. It finds csc.py in the current directory and since .py is a registered extension, that's what gets executed.
The solution is to rename your python file or call out 'csc.exe' explicitly.
I am building Froyo, is it possible that during building, make/python can output the file and the command it is calling right now.
For example, in one of the Android.mk, there is a line, says,
echo build success.
On the monitor it will show "build success",
what I want is that in addition, it shows
"Android.mk line 20: echo build success".
Is it possible?
The message parser of the android make comment accepts info and warning tags in your Android.mk.
For example, If you want to print the value of an internal variable:
LOCAL_CFLAGS := -DHAVE_ERRNO_H -g
$(info value of LOCAL_CFLAGS is: $(LOCAL_CFLAGS))
the info tells the compiler to print info debug output.
You can do the same with warning and error
$(warning value of LOCAL_CFLAGS is: $(LOCAL_CFLAGS))
would print a highlighted warning message
and
$(error value of LOCAL_CFLAGS is: $(LOCAL_CFLAGS))
would print the message and stop the build.
I've just experienced an odd effect of using $(info) when compiling a java+C++ Android application:
I used $info) to output some informations about conditional compiling in the Android.mk of the main application and when trying to debug the native part of the program, using ndk-gdb, it failed because apparently the output of $(info) is read by the ndk-gdb script (using the get_build_var() and get_build_var_for_abi() functions).
THe result is that the ndk-gdb script is not executed properly.