I have a swagger generated (python-flask) web-server, which supports a variety of requests, which are fed by requests sent from an instance of QNetworkAccessManager (GET, PUT, POST).
They actually work like a charm, so I assume, that I got the main usage right, more or less.
What doesn't work though, is file upload using POST request. I actually tried several variants:
Providing the file content with QNetworkAccessManager::send,
Providing the file-pointer with QNetworkAccessManager::send (yes, I made sure, that the file is open and valid during the complete operation and cleaned up afterwards),
Using QMultiPart (see example below),
Playing with the request parameters didn't help either
But whatever I try, the request is recorded in the web-server, but connexion.request.files.getlist('file') is empty.
On the other hand, using curl -X POST --header 'Content-Type: multipart/form-data' --header 'Accept: application/json' -F "file=#/path/to/file/image.jpg" {"type":"formData"} '$myURL' makes it [<FileStorage: 'image.jpg' ('image/jpeg')>].
void uploadImage(QUrl const& url, std::filesystem::path const& filepath)
{
QNetworkRequest request {url};
request.setRawHeader("Content-Type", "multipart/form-data");
QHttpMultiPart *multiPart = new QHttpMultiPart(QHttpMultiPart::FormDataType);
QHttpPart imagePart;
imagePart.setHeader(QNetworkRequest::ContentDispositionHeader, QVariant("form-data; name=\"" + QString::fromStdString(filepath.filename().native()) + "\"; filename=\"" + QString::fromStdString(filepath.native()) + "\""));
imagePart.setHeader(QNetworkRequest::ContentTypeHeader, QVariant("image/jpeg"));
QFile *file = new QFile(QString::fromStdString(filepath.native()));
if (!file->open(QIODevice::ReadOnly)) {
std::cout << "could not open file" << std::endl;
}
QByteArray fileContent(file->readAll());
imagePart.setBody(fileContent);
multiPart->append(imagePart);
request.setHeader(QNetworkRequest::ContentLengthHeader, file->size());
auto reply = _manager->post(request, multiPart);
file->setParent(reply);
connect(reply, &QNetworkReply::finished, reply, &QNetworkReply::deleteLater);
}
QProcess using the curl command wasn't that happy and me neither, because, even though I'm running the code currently on a Linux machine, I don't want to restrict to Linux only.
Are there any suggestions on how to use Qt-Network in this context?
Related
I'm currently modernizing a request from a bash curl to a python request. Here is before:
curl -X GET --header 'Accept: text/csv' 'https://URL?afterDate=1650914642000&beforeDate=2000000000000'
Here is the new process:
import requests
import pandas as pd
import io
args = {
'params': {'afterDate': 1650914642000, 'beforeDate': 2000000000000},
'data': {},
'headers': {'Accept': 'text/csv'},
'timeout': 3600
}
req = 'URL'
r = requests.request("GET", req, **args)
str_buffer = io.StringIO()
str_buffer.write(r.text)
str_buffer.seek(0)
df = pd.read_csv(
str_buffer,
escapechar="\\",
sep=",",
quotechar='"',
low_memory=False
)
Surprisingly, this is not working. The curl works and the requests version does not. Talking to the engineers on the other end, they are seeing all green lights and a bit of latency, but nothing very abnormal. As far as I can tell, the request just sits and does not receive the data the endpoint returns.
Additional Context:
This process is for catching a daily delta load, so the 2000000000000 param is just an arbitrarily high value and even for a single minute of time I cannot return data. The endpoint is a java based api hitting a mongo source, and there is a fair amount of latency, even on the curl request.
I'm at a loss of how to dig deeper than "It's not working on my machine", if anyone knows of anything that could help, help a guy out!
EDIT:
After 30minutes of running (it should take about 10-15 min), when I keyboardInterrupt I get this trace (this is the end):
File "/home/rriley2/anaconda3/envs/env_etl/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1839, in recv_into
result = _lib.SSL_read(self._ssl, buf, nbytes)
KeyboardInterrupt
I have ran tika server in my machine and call api using terminal which is working fine. I am able to extract text from image and pdf. But, I want to implement the api call in my python application.
curl -T price.xls http://localhost:9998/tika --header "Accept: text/plain"
Above is api call that i have to make. I can run this in my terminal and works fine but how to implement in python application. I have installed and tried requests.
API_URL = 'http://localhost:9998/tika'
APP_ROOT = os.path.dirname(os.path.abspath(__file__))
tika_client = TikaApp(file_jar=join(APP_ROOT,'../tika-app-1.19.jar'))
data = {
"url": join(APP_ROOT,'../static/image/a.pdf')
}
response = requests.put(API_URL, data)
print(response.content)
Any help will be appreciate. Thank you :)
error output
INFO tika (application/x-www-form-urlencoded)
WARN tika: Text extraction failed
org.apache.tika.exception.TikaException: Unexpected RuntimeException from org.apache.tika.server.resource.TikaResource$1#475b0e2
at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:282)
at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:143)
at org.apache.tika.server.resource.TikaResource.parse(TikaResource.java:402)
at org.apache.tika.server.resource.TikaResource$5.write(TikaResource.java:513)
at org.apache.cxf.jaxrs.provider.BinaryDataProvider.writeTo(BinaryDataProvider.java:177)
at org.apache.cxf.jaxrs.utils.JAXRSUtils.writeMessageBody(JAXRSUtils.java:1391)
at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.serializeMessage(JAXRSOutInterceptor.java:246)
at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.processResponse(JAXRSOutInterceptor.java:122)
at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.handleMessage(JAXRSOutInterceptor.java:84)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
at org.apache.cxf.interceptor.OutgoingChainInterceptor.handleMessage(OutgoingChainInterceptor.java:90)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:267)
at org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(JettyHTTPDestination.java:247)
at org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPHandler.java:79)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:205)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:531)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.ws.rs.WebApplicationException: HTTP 415 Unsupported Media Type
at org.apache.tika.server.resource.TikaResource$1.parse(TikaResource.java:128)
at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:280)
... 37 more
ERROR Problem with writing the data, class org.apache.tika.server.resource.TikaResource$5, ContentType: text/plain
You need to define the data(payload), header.
url = 'http://localhost:9998/tika/......'
headers = {"Accept: text/plain"}
response = requests.put(url,data = data, headers=headers)
Have glance at this Making a request to a RESTful API using python
I am following this link in order to make an HTTP POST to an Eventhub in Azure. the error I am getting is 401 40103: Invalid authorization token signature. According to Azure, the POST should have the following format:
POST https://your-namespace.servicebus.windows.net/your-event-hub/partitions/0/messages?timeout=60&api-version=2014-01 HTTP/1.1
Authorization: SharedAccessSignature sr=your-namespace.servicebus.windows.net&sig=your-sas-key&se=1403736877&skn=RootManageSharedAccessKey
Content-Type: application/atom+xml;type=entry;charset=utf-8
Host: your-namespace.servicebus.windows.net
Regarding the Authorization header, I have several questions:
My secret key (sig) has an equal sign, should I substitute it by
%3d?
I am currently doing also the POST operation with the
example scripts in Python and C from Azure. In these examples, is
only required to introduce the endpoint with all the credentials and
the post/get operations works fine. Is it possible to perform the
put/get operations introducing directly the endpoint in the api
rest, or obtaining the authorization header that is performed bellow
the python or c code?
Thanks.
I'm able to make it work by creating a SAS policy (nssend below) on the namespace, not on the Event Hub.
$ curl -X POST -i "https://breakingnews.servicebus.windows.net/sucheventsmuchwow/messages" \
-H "Authorization: SharedAccessSignature sr=https%3A%2F%2Fbreakingnews.servicebus.windows.net%2Fsucheventsmuchwow%2Fmessages&sig=SAS_SIG_GOES_HERE&se=1536733370&skn=nssend" \
-H "Content-Type: application/json" \
--data-ascii "{ \"message\": \"So many events, so little time.\" }"
HTTP/1.1 201 Created
Server: Microsoft-HTTPAPI/2.0
...
So that works.
However, i'm getting a HTTP 401, just like you are, when using an Event Hub level SAS policy to generate the signature (as opposed to a namespace level policy).
This is what i used to generate the SAS token —
// Make a SAS token
// See https://learn.microsoft.com/en-us/rest/api/eventhub/generate-sas-token
// Appologies for JavaScript
// npm install moment
const moment = require('moment');
const crypto = require('crypto');
function create_sas_token(uri, key_name, key)
{
// Token expires in one hour
var expiry = moment().add(7, 'days').unix();
var string_to_sign = encodeURIComponent(uri) + '\n' + expiry;
var hmac = crypto.createHmac('sha256', key);
hmac.update(string_to_sign);
var signature = hmac.digest('base64');
var token = 'SharedAccessSignature sr=' +
encodeURIComponent(uri) +
'&sig=' + encodeURIComponent(signature) +
'&se=' + expiry + '&skn=' + key_name;
return token;
}
let token = create_sas_token('https://breakingnews.servicebus.windows.net/sucheventsmuchwow/messages', 'MySendPolicy', 'MySendPolicyPrimaryKey=');
console.log(token);
UPDATE
Thanks to Clemens Vasters —
Try omitting "/messages"— Clemens Vasters 📨, Messenger (#clemensv) September 5, 2018
Your string to sign (resource URI) should omit /messages, e.g.
create_sas_token('https://breakingnews.servicebus.windows.net/sucheventsmuchwow',
'MyEventHubLevelPolicy', 'hUbPriMAry/KEy=');
then craft your request as follows —
$ curl -X POST -i "https://breakingnews.servicebus.windows.net/sucheventsmuchwow/messages" \
-H "Authorization: SharedAccessSignature sr=https%3A%2F%2Fbreakingnews.servicebus.windows.net%2Fsucheventsmuchwow&sig=DONT_INCLUDE_/MESSAGES_IN_STRING_TO_SIGN&se=1536757127&skn=MyEventHubLevelPolicy" \
-H "Content-Type: application/json" \
--data-ascii "{ \"message\": \"SAS signed with Event Hub level policy\" }"
HTTP/1.1 201 Created
Server: Microsoft-HTTPAPI/2.0
...
TL;DR:
Your POST URL should include the trailing /messages, but the string to sign (resource URI) should not. Always. Regardless if using Namespace or Hub-scoped policy.
I am trying to launch a Jenkins parametrized job from a python script. Due to environment requirements, I can't install python-jenkins. I am using raw requests module.
This job I am trying to launch has three parameters:
string (let's call it payload)
string (let's call it target)
file (a file, optional)
I've searched and search, without any success.
I managed to launch the job with two string parameters by launching:
import requests
url = "http://myjenkins/job/MyJobName/buildWithParameters"
target = "http://10.44.542.62:20000"
payload = "{payload: content}"
headers = {"Content-Type": "application/x-www-form-urlencoded"}
msg = {
'token': 'token',
'payload': [ payload ],
'target': [ target ],
}
r = requests.post(url, headers=headers, data=msg)
However I am unable to send a file and those arguments in single request.
I've tried requests.post file argument and failed.
It turns out it is impossible to send both data and file in a single request via HTTP.
import jenkinsapi
from jenkinsHandler import JenkinsHandler
in your python script
Pass parameters to buildJob() , (like < your JenkinsHandler object name>.buildJob())
JenkinsHandler module has functions like init() , buildJob(), isRunning() which helps in triggering the build
Here is an example:
curl -vvv -X POST http://127.0.0.1:8080/jenkins/job/jobname/build
--form file0='#/tmp/yourfile'
--form json='{"parameter": [{"name":"file", "file":"file0"}]}'
--form json='{"parameter": [{"name":"payload", "value":"123"}]
--form json='{"parameter": [{"name":"target", "value":"456"}]}'
I want to create a ticket with attachment in freshdesk api. I am able to create a ticket without attachment. This is my sample code:
post_dict = {
'helpdesk_ticket': {
'description': "Testing Code sample 3",
'subject': "example7",
'email': "example7#example.com",
'priority': 2,
'status': 2,
},
}
headers = {'Content-Type': 'application/json'}
r = requests.post(FRESHDESK_URL + '/helpdesk/tickets.json',
auth=(FRESHDESK_API_KEY, "X"),
headers=headers,
data=json.dumps(post_dict),
)
raw_input(r.status_code)
raw_input(r.content)
This is for just creating ticket in Freshdesk. Now using the same post_dict i would like to create tickets with attachments. Any suggestions about how I can achieve this using this json request method or any other methods are welcome.
Creating a ticket with an attachment requires a multi-part form submission. Unfortunately that means that the request is very different to a simple request without an attachment.
Each field needs to be written to the request as a separate form with a 'boundary' before it and a character return after it.
Then, each attachment should be written to the request, again with a boundary before the attachment is written and a character return added after.
At the end of the response, a final boundary must be written. This is the same as the string used for the boundary but also includes 2 dashes ( -- ) before and after the boundary to signify it as final. Without the final boundary, FreshDesk gives a 500 internal server error since something changed in their api a few weeks back (it used to accept a non-final boundary at the end).
For Send attachment, the Content-Type must be in multipart/form-data.The sample cURL is may helps you.
curl -v -u yourmail#gmail.com:yourpassword -F
"attachments[]=#/path/to/attachment1.ext" -F
"attachments[]=#/path/to/attachment2.ext" -F "email=example#example.com" -F
"subject=Ticket Title" -F "description=this is a sample ticket" -X POST
'https://yoururl.freshdesk.com/api/v2/tickets'