I'm trying to program a small HTTP local proxy server to run on my machine and run some tests.
My server currently runs perfectly and serves the requests fine.
However, when I try to analyse the packer - I get a problem.
I'm searching for the tag "" in my packets, and print a message to a log when I find it.
It works on a very limited number of websites, while on the other, like StackOverflow for example, it doesn't.
Do I need to some sort of decoding before I search for the word in the received data? If so - which decoding? How do I recode the data to serve to the browser?
Here's my code for the searching and replacing:
data = i.recv(8192)
if data:
if "<head>" in data:
print "Found Head Tag."
The above code is a simple python code to retrieve the data from the socket, save it to the data object, and search for the wanted tag. As I said, it works on very few websites, and not on the others.
Many webservers use compression to lower bandwidth usage.
You will need to check HTTP headers for Content-Encoding and apply the required operations (i.e. gzip decompression) to get the plain text.
Related
I have a TCP server written in SwiftNIO, based on this documentation.
I want my client to be written in python from which I can send multiple JSON strings & can receive similar/different multiple JSON string(s) as a response periodically for a few minutes.
In which format do I need to convert those JSON strings from the python client & how do I get the same JSON string on the SwiftNIO server (and vice versa)?
If I were you, I'd use HTTP using the Vapor web server and any Python HTTP library such as requests. If you do that, then your job will be pretty straightforward. The Vapor community is also super helpful in their Discord chat.
If you really want to do this in a low-level library like SwiftNIO then that's of course possible but you'll need to design a so called "wire protocol" for the framing (ie. when does one JSON message start and end). SwiftNIO is very well equipped for these things but you'll likely need to learn a bunch of things.
You could for example use NIO Extras' LineBasedFrameDecoder and send each JSON (make sure it doesn't contain newlines) followed by a \n. Or you could say that you prepend the JSON by say a 32 bit length field (which you could decode using the LengthFieldBasedFrameDecoder. There are many options...
You could also implement JSON-RPC and you could get some inside in this example which is also explained in this talk.
I am using a nodejs express server that sends data to a python client. The nodejs express server sends data using res.send() function and the returned data is huge and sometimes when I am processing the data at the python side using response.json() function I get this error. Here are is my understanding of the error , either the python side the entire request was not read or the node server truncated the data when it reached a maximum size.
Here are my questions:
1. Am I supposed to use res.json() instead of res.send() ? Since to my understanding res.json() also calls res.send() inturn.
2. Am I supposed to stream the response data in python ? is that a good option? or is it even an option ?
3. Is this a configuration issue ? As I understand there are ways to configure the nginx server that is connecting these microservices to limit the amount of data transferred?
4. Is there way to ensure that data transmitted from the node server always contains the complete json. Liking parseing the body before sending it.
I am beginner and If I had made any mistakes or suggestions in the questions, do point them out.
I am writing a simple SOAP client application in Python.
WSDL file can be found here: https://clients.nationalmailing.com.au/ServiceTest/OrderService.svc?wsdl
Unfortunately the server declared usage of wsHttpBinding in its WSDL file and I had to learn how many troubles it brings to not-.NET developers.
I have working C# code (and it is pretty simple there) and used Fiddler to capture the traffic and analyze messages. Now I know the structure to follow. Client sends 2 subsequental messages.
I managed to create and send first request and receive a response from the server. BUT second request is a way more complex. I have found a library signxml which helped me to create <Signature> structure with all the fields that should present (as per captured traffic).
But the server continues to answer with "Error 500: An error occurred when verifying security for the message."
I realized that in the first message I put just random values for the following structure:
<s:Body>
<trust:RequestSecurityToken xmlns:trust="http://docs.oasis-open.org/ws-sx/ws-trust/200512">
<trust:TokenType>http://docs.oasis-open.org/ws-sx/ws-secureconversation/200512/sct</trust:TokenType>
<trust:RequestType>http://docs.oasis-open.org/ws-sx/ws-trust/200512/Issue</trust:RequestType>
<trust:Entropy>
<trust:BinarySecret
u:Id="uuid-0649fd7a-9ae2-4f9f-964c-e3aa5d68e8cd-1"
Type="http://docs.oasis-open.org/ws-sx/ws-trust/200512/Nonce">h/MaeQVSL5Br30Hnt/SAl274flYfZVZyx2Fri9zNuEY=</trust:BinarySecret>
</trust:Entropy>
<trust:KeySize>256</trust:KeySize>
</trust:RequestSecurityToken>
</s:Body>
The value of BinarySecret is just a random string encoded with Base64. I think this should be an issue on this stage. I also do not use the same parameters from server's response.
Could anyone explain how should I use Entropy.BinarySecret - should it take part in the calculations of Signature and how it is used?
Answering my own question. Yes, the issue was in improper usage of Entropy parameter.
To sign the message you need to generate a key, it consists of two parts (client entropy and server's entropy). They get combined with P_SHA1 algorithm into a key.
To anyone who find this post in the future: for Python have a look on signxml library and section 4 of ws-trust spec.
We have two servers (client-facing, and back-end database) between which we would like to transfer PDFs. Here's the data flow:
User requests PDF from website.
Site sends request to client-server.
Client server requests PDF from back-end server (different IP).
Back-end server sends PDF to client server.
Client server sends PDF to website.
1-3 and 5 are all good, but #4 is the issue.
We're currently using Flask requests for our API calls and can transfer text and .csv easily, but binary files such as PDF are not working.
And no, I don't have any code, so take it easy on me. Just looking for a suggestion from someone who may have come across this issue.
As you said you have no code, that's fine, but I can only give a few suggestions.
I'm not sure how you're sending your files, but I'm assuming that you're using pythons open function.
Make sure you are reading the file as bytes (e.g. open('<pdf-file>','rb'))
Cut the file up into chunks and send it as one file, this way it doesn't freeze or get stuck.
Try smaller PDF files, if this works definitely try suggestion #2.
Use threads, you can multitask with them.
Have a download server, this can save memory and potentially save bandwidth. Also it also lets you skip the PDF send back, from flask.
Don't use PDF files if you don't have to.
Use a library to do it for you.
Hope this helps!
I wanted to share my solution to this, but give credit to #CoolqB for the answer. The key was including 'rb' to properly read the binary file and including the codecs library. Here are the final code snippets:
Client request:
response = requests.get('https://www.mywebsite.com/_api_call')
Server response:
f = codecs.open(file_name, 'rb').read()
return f
Client handle:
with codecs.open(file_to_write, 'w') as f:
f.write(response.content)
f.close()
And all is right with the world.
I apologise if this is a daft question. I'm currently writing against a Django API (which I also maintain) and wish under certain circumstances to be able to generate multiple partial responses in the case where a single request yields a large number of objects, rather than sending the entire JSON structure as a single response.
Is there a technique to do this? It needs to follow a standard such that client systems using different request libraries would be able to make use of the functionality.
The issue is that the client system, at the point of asking, does not know the number of objects that will be present in the response.
If this is not possible, then I will have to chain requests on the client end - for example, getting the first 20 objects & if the response suggests there will be more, requesting the next 20 etc. This approach is an OK work-around, but any subsequent requests rely on the previous response. I'd rather ask once and have some kind of multi-part response.
As far as I know, No you can't send Multipart http response not yet atleast. Multipart response is only valid in http requests. Why? Because no browser as I know of completely supports this.
Firefox 3.5: Renders only the last part, others are ignored.
IE 8: Shows all the content as if it were text/plain, including the boundaries.
Chrome 3: Saves all the content in a single file, nothing is rendered.
Safari 4: Saves all the content in a single file, nothing is rendered.
Opera 10.10: Something weird. Starts rendering the first part as plain/text, and then clears everything. The loading progress bar hangs on 31%.
(Data credits Diego Jancic)