Develop Reference

AWS Glue error - Invalid input provided while running python shell program

I have Glue job, a python shell code. When I try to run it I end up getting the below error.
Job Name : xxxxx Job Run Id : yyyyyy failed to execute with exception Internal service error : Invalid input provided
It is not specific to code, even if I just put
import boto3
print('loaded')
I am getting the error right after clicking the run job option. What is the issue here?

It happend to me but the same job is working on a different account.
AWS documentation is not really explainative about this error:
The input provided was not valid.
I doubt this is an Amazon issue as mentionned #Quartermass

Same issue here in eu-west-2 yesterday, working now. This was only happening with Pythonshell jobs, not Pyspark ones, and job runs weren't getting as far as outputting any log streams. I can only assume it was an AWS issue they've now fixed and not issued a service announcement for.

I think Quatermass is right, the jobs started working out of the blue the next day without any changes.

I too received this super helpful error message.
What worked for me was explicitly setting properties like worker type, number of workers, Glue version and Python version.
In Terraform code:
resource "aws_glue_job" "my_job" {
name = "my_job"
role_arn = aws_iam_role.glue.arn
worker_type = "Standard"
number_of_workers = 2
glue_version = "4.0"
command {
script_location = "s3://my-bucket/my-script.py"
python_version = "3"
}
default_arguments = {
"--enable-job-insights" = "true",
"--additional-python-modules" : "boto3==1.26.52,pandas==1.5.2,SQLAlchemy==1.4.46,requests==2.28.2",
}
}
Update
After doing some more digging, I realised that what I needed was a Python shell script Glue job, not an ETL (Spark) job. By choosing this flavour of job, setting the Python version to 3.9 and "ticking the box" for Glue's pre-installed analytics libraries, my script, incidentally, had access to all the libraries I needed.
My Terraform code ended up looking like this:
resource "aws_glue_job" "my_job" {
name = "my-job"
role_arn = aws_iam_role.glue.arn
glue_version = "1.0"
max_capacity = 1
connections = [
aws_glue_connection.redshift.name
]
command {
name = "pythonshell"
script_location = "s3://my-bucket/my-script.py"
python_version = "3.9"
}
default_arguments = {
"--enable-job-insights" = "true",
"--library-set" : "analytics",
}
}
Note that I have switched to using Glue version 1.0. I arrived at this after some trial and error, and could not find this explicitly stated as the compatible version for pythonshell jobs… but it works!

Well, in my case, I get this error from time to time without any clear reason. The only thing that seems to cause the issue, is modifying some job parameter and saving the modifications. As soon as I save and try to execute the job, I usually get this error and, the only way to solve the issue, is destroying the job and, then, re-creating it again. Does anybody solved this issue by other means? As I saw in the accepted answer, the job simply begun to work again wthout any manual action, giving an understanding that the problem was a bug in AWS that was corrected.

I was facing a similar issue. I was invoking my job from a workflow. I could solve it by adding WorkerType, GlueVersion, NumberOfWorkers to the job before adding the job to the workflow. I could see it consistently fail before and succeed after this addition.

Azure CosmosDB why isn't the 'id' field automatically added when inserting into Azure CosmosDB?

I have recently started getting this error when I debug my python function app: (BadRequest) Message: {"Errors":["One of the specified inputs is invalid"]}
The function app works perfectly well for my coworkers and they are not getting this error.
The error occurs whenever I attempt to access the azure cosmos db.
Sample
item = { "firstname": "John", "lastname": "Doe"}
container.upsert_item(item)
The issue seems to be that insert into the database is missing the "id" field. If I add this field to the item before calling upsert_item(item) it works correctly.
Why did I not originally need to do this (coworkers don't need to) and what setting did I change to stop it from auto adding the id field?
I am using visual studio code, coding in python, installed the azure tools extension. I have tried deleting the code from disk and re-cloning it, removing all the extensions and reinstalling them, and I even completely uninstalled visual studio code. Whatever setting changed is still there.
EDIT
Added the container properties
Container properties {'id': 'person', 'indexingPolicy': {'indexingMode': 'consistent', 'automatic': True, 'includedPaths': [{'path': '/*'}], 'excludedPaths': [{'path': '/"_etag"/?'}]}, 'partitionKey': {'paths': ['/lastname'], 'kind': 'Hash'}, 'uniqueKeyPolicy': {'uniqueKeys': []}, 'conflictResolutionPolicy': {'mode': 'LastWriterWins', 'conflictResolutionPath': '/_ts', 'conflictResolutionProcedure': ''}, 'geospatialConfig': {'type': 'Geography'}, 'analyticalStorageTtl': -1, '_rid': 'O-pDAPQSY5Y=', '_ts': 1654191756, '_self': 'dbs/O-pDAA==/colls/O-pDAPQSY5Y=/', '_etag': '"00003b31-0000-0100-0000-6298f68c0000"', '_docs': 'docs/', '_sprocs': 'sprocs/', '_triggers': 'triggers/', '_udfs': 'udfs/', '_conflicts': 'conflicts/'}

Found the answer to my issue.
I had updated azure cosmos db to the latest version (4.3.0) globaly (which is why it only affected me, should have put it in the requirements.txt instead) which does not allow for the automatic creation of ids.
There is a single line change (from 4.2.0 to 4.3.0) in the upsert_item(...) function that is difficult to spot (worded very similarly) when you are looking at a wall of code.
Azure-Cosmos-4.3.0 github link to the line
Annoyingly this line is hard coded. Why they decided to prevent the user from auto generating ids is beyond me.

Python JIRA jira.issue slow performance

I am trying to retrieve from JIRA tree graph of parent child issues
epic->story->task using python3.10 jira=3.1.1
for project with 400 issues it takes minutes to retrieve the result.
is there a way how to improve this code for better performance
The result is displayed in flask frontend web and response time 2minutes is not pleasant for users
jira = JIRA(options={'server': self.url, 'verify': False}, basic_auth=(self.username, self.password))
for singleIssue in jira.search_issues(jql_str=querystring, maxResults=False):
issue = jira.issue(singleIssue.key)
links = issue.fields.issuelinks
for link in links:
if hasattr(link, 'inwardIssue'):
item['inward_issue'] = link.inwardIssue.key
if hasattr(link, 'outwardIssue'):
item['outward_issue'] = link.outwardIssue.key
item['typename'] = link.type.nam

I realised this is moreless matter of flask and long queries handling.
It should be solved either by asynchronous calls or queuing.

Limit the fields you are requesting via the "fields=[ 'key' ]" parameter to include only those which you need.
In your example you are fetching the whole issue twice: once from the .search_issues() iterator every time round the loop (as expected), and (unnecessarily) again in the .issue() call on the following line.

How to change username of job in print queue using python & win32print

I am trying to change the user of a print job in the queue, as I want to create it on a service account but send the job to another users follow-me printing queue. I'm using the win32 module in python. Here is an example of my code:
from win32 import win32print
JOB_INFO_LEVEL = 2
pclExample = open("sample.pcl")
printer_name = win32print.GetDefaultPrinter()
hPrinter = win32print.OpenPrinter(printer_name)
try:
jobID = win32print.StartDocPrinter(hPrinter, 1, ("PCL Data test", None, "RAW"))
# Here we try to change the user by extracting the job and then setting it again
jobInfoDict = win32print.GetJob(hPrinter, jobID , JOB_INFO_LEVEL )
jobInfoDict["pUserName"] = "exampleUser"
win32print.SetJob(hPrinter, jobID , JOB_INFO_LEVEL , jobInfoDict , win32print.JOB_CONTROL_RESUME )
try:
win32print.StartPagePrinter(hPrinter)
win32print.WritePrinter(hPrinter, pclExample)
win32print.EndPagePrinter(hPrinter)
finally:
win32print.EndDocPrinter(hPrinter)
finally:
win32print.ClosePrinter(hPrinter)
The problem is I get an error at the win32print.SetJob() line. If JOB_INFO_LEVEL is set to 1, then I get the following error:
(1804, 'SetJob', 'The specified datatype is invalid.')
This is a known bug to do with how the C++ works in the background (Issue here).
If JOB_INFO_LEVEL is set to 2, then I get the following error:
(1798, 'SetJob', 'The print processor is unknown.')
However, this is the processor that came from win32print.GetJob(). Without trying to change the user this prints fine, so I'm not sure what is wrong.
Any help would be hugely appreciated! :)
EDIT:
Using Python 3.8.5 and Pywin32 303

At the beginning I thought it was a misunderstanding (I was also a bit skeptical about the bug report), mainly because of the following paragraph (which apparently seems to be wrong) from [MS.Docs]: SetJob function (emphasis is mine):
The following members of a JOB_INFO_1, JOB_INFO_2, or JOB_INFO_4 structure are ignored on a call to SetJob: JobId, pPrinterName, pMachineName, pUserName, pDrivername, Size, Submitted, Time, and TotalPages.
But I did some tests and ran into the problem. The problem is as described in the bug: filling JOB_INFO_* string members (which are LPTSTRs) with char* data.
Submitted [GitHub]: mhammond/pywin32 - Fix: win32print.SetJob sending ANSI to UNICODE API (and none of the 2 errors pops up). It was merged to main on 220331.
When testing the fix, I was able to change various properties of an existing job, I was amazed that it didn't have to be valid data (like below), I'm a bit curious to see what would happen when the job would be executed (as now I don't have a connection to a printer):
Change pUserName to str(random.randint(0, 10000)) to make sure it changes on each script run (PrintScreens taken separately and assembled in Paint):
Ways to go further:
Wait for a new PyWin32 version (containing this fix) to be released. This is the recommended approach, but it will also take more time (and it's unclear when it will happen)
Get the sources, either:
from main
from b303 (last stable branch), and apply the (above) patch(1)
build the module (.pyd) and copy it in the PyWin32's site-packages directory on your Python installation(s). Faster, but it requires some deeper knowledge, and maintenance might become a nightmare
Footnotes
#1: Check [SO]: Run / Debug a Django application's UnitTests from the mouse right click context menu in PyCharm Community Edition? (#CristiFati's answer) (Patching UTRunner section) for how to apply patches (on Win).

How can I consume a WSDL (SOAP) web service in Python?

I want to use a WSDL SOAP based web service in Python. I have looked at the Dive Into Python code but the SOAPpy module does not work under Python 2.5.
I have tried using suds which works partly, but breaks with certain types (suds.TypeNotFound: Type not found: 'item').
I have also looked at Client but this does not appear to support WSDL.
And I have looked at ZSI but it looks very complex. Does anyone have any sample code for it?
The WSDL is https://ws.pingdom.com/soap/PingdomAPI.wsdl and works fine with the PHP 5 SOAP client.

I would recommend that you have a look at SUDS
"Suds is a lightweight SOAP python client for consuming Web Services."

There is a relatively new library which is very promising and albeit still poorly documented, seems very clean and pythonic: python zeep.
See also this answer for an example.

I recently stumbled up on the same problem. Here is the synopsis of my solution:
Basic constituent code blocks needed
The following are the required basic code blocks of your client application
Session request section: request a session with the provider
Session authentication section: provide credentials to the provider
Client section: create the Client
Security Header section: add the WS-Security Header to the Client
Consumption section: consume available operations (or methods) as needed
What modules do you need?
Many suggested to use Python modules such as urllib2 ; however, none of the modules work-at least for this particular project.
So, here is the list of the modules you need to get.
First of all, you need to download and install the latest version of suds from the following link:
pypi.python.org/pypi/suds-jurko/0.4.1.jurko.2
Additionally, you need to download and install requests and suds_requests modules from the following links respectively ( disclaimer: I am new to post in here, so I can't post more than one link for now).
pypi.python.org/pypi/requests
pypi.python.org/pypi/suds_requests/0.1
Once you successfully download and install these modules, you are good to go.
The code
Following the steps outlined earlier, the code looks like the following:
Imports:
import logging
from suds.client import Client
from suds.wsse import *
from datetime import timedelta,date,datetime,tzinfo
import requests
from requests.auth import HTTPBasicAuth
import suds_requests
Session request and authentication:
username=input('Username:')
password=input('password:')
session = requests.session()
session.auth=(username, password)
Create the Client:
client = Client(WSDL_URL, faults=False, cachingpolicy=1, location=WSDL_URL, transport=suds_requests.RequestsTransport(session))
Add WS-Security Header:
...
addSecurityHeader(client,username,password)
....
def addSecurityHeader(client,username,password):
security=Security()
userNameToken=UsernameToken(username,password)
timeStampToken=Timestamp(validity=600)
security.tokens.append(userNameToken)
security.tokens.append(timeStampToken)
client.set_options(wsse=security)
Please note that this method creates the security header depicted in Fig.1. So, your implementation may vary depending on the correct security header format provided by the owner of the service you are consuming.
Consume the relevant method (or operation) :
result=client.service.methodName(Inputs)
Logging:
One of the best practices in such implementations as this one is logging to see how the communication is executed. In case there is some issue, it makes debugging easy. The following code does basic logging. However, you can log many aspects of the communication in addition to the ones depicted in the code.
logging.basicConfig(level=logging.INFO)
logging.getLogger('suds.client').setLevel(logging.DEBUG)
logging.getLogger('suds.transport').setLevel(logging.DEBUG)
Result:
Here is the result in my case. Note that the server returned HTTP 200. This is the standard success code for HTTP request-response.
(200, (collectionNodeLmp){
timestamp = 2014-12-03 00:00:00-05:00
nodeLmp[] =
(nodeLmp){
pnodeId = 35010357
name = "YADKIN"
mccValue = -0.19
mlcValue = -0.13
price = 36.46
type = "500 KV"
timestamp = 2014-12-03 01:00:00-05:00
errorCodeId = 0
},
(nodeLmp){
pnodeId = 33138769
name = "ZION 1"
mccValue = -0.18
mlcValue = -1.86
price = 34.75
type = "Aggregate"
timestamp = 2014-12-03 01:00:00-05:00
errorCodeId = 0
},
})

Zeep is a decent SOAP library for Python that matches what you're asking for: http://docs.python-zeep.org

Right now (as of 2008), all the SOAP libraries available for Python suck. I recommend avoiding SOAP if possible. The last time we where forced to use a SOAP web service from Python, we wrote a wrapper in C# that handled the SOAP on one side and spoke COM out the other.

I periodically search for a satisfactory answer to this, but no luck so far. I use soapUI + requests + manual labour.
I gave up and used Java the last time I needed to do this, and simply gave up a few times the last time I wanted to do this, but it wasn't essential.
Having successfully used the requests library last year with Project Place's RESTful API, it occurred to me that maybe I could just hand-roll the SOAP requests I want to send in a similar way.
Turns out that's not too difficult, but it is time consuming and prone to error, especially if fields are inconsistently named (the one I'm currently working on today has 'jobId', JobId' and 'JobID'. I use soapUI to load the WSDL to make it easier to extract endpoints etc and perform some manual testing. So far I've been lucky not to have been affected by changes to any WSDL that I'm using.

It's not true SOAPpy does not work with Python 2.5 - it works, although it's very simple and really, really basic. If you want to talk to any more complicated webservice, ZSI is your only friend.
The really useful demo I found is at http://www.ebi.ac.uk/Tools/webservices/tutorials/python - this really helped me to understand how ZSI works.

If you're rolling your own I'd highly recommend looking at http://effbot.org/zone/element-soap.htm.

SOAPpy is now obsolete, AFAIK, replaced by ZSL. It's a moot point, because I can't get either one to work, much less compile, on either Python 2.5 or Python 2.6

#!/usr/bin/python
# -*- coding: utf-8 -*-
# consume_wsdl_soap_ws_pss.py
import logging.config
from pysimplesoap.client import SoapClient
logging.config.dictConfig({
'version': 1,
'formatters': {
'verbose': {
'format': '%(name)s: %(message)s'
}
},
'handlers': {
'console': {
'level': 'DEBUG',
'class': 'logging.StreamHandler',
'formatter': 'verbose',
},
},
'loggers': {
'pysimplesoap.helpers': {
'level': 'DEBUG',
'propagate': True,
'handlers': ['console'],
},
}
})
WSDL_URL = 'http://www.webservicex.net/stockquote.asmx?WSDL'
client = SoapClient(wsdl=WSDL_URL, ns="web", trace=True)
client['AuthHeaderElement'] = {'username': 'someone', 'password': 'nottelling'}
#Discover operations
list_of_services = [service for service in client.services]
print(list_of_services)
#Discover params
method = client.services['StockQuote']
response = client.GetQuote(symbol='GOOG')
print('GetQuote: {}'.format(response['GetQuoteResult']))