Develop Reference

Ruby net/http GET requests with empty body [duplicate]

This question already has answers here:
Ruby - net/http - following redirects
(6 answers)
Closed 17 days ago.
I'm currently simply trying to get a simple GET request working in Ruby, however, I'm seeing some strange behavior.
I have an Open Web Analytics application running with Docker and it is reachable at http://127.0.0.1:8080/.
I can reach the login site and everything works fine.
Now I want to do a GET request with Ruby to analyze the body of that request but I cannot get it to work, in other languages like Python or simple GET requests over the terminal it works fine. Why not with Ruby?
Here is my very basic Ruby code:
require 'net/http'
url = 'http://127.0.0.1:8080/'
uri = URI(url)
session = Net::HTTP.new(uri.host, uri.port)
response = session.get(uri.request_uri)
puts response.body
Which doesn't output anything. If I look into the NGINX logs from the container, I can see the request being made but there is no further redirection as with the other methods (see below).
172.23.0.1 - - [02/Feb/2023:20:02:59 +0000] "GET / HTTP/1.1" 302 5 "-" "Ruby" "-" 0.088 0.088 . -
If I do a simple GET over the terminal, it works:
GET http://127.0.0.1:8080/
will output the correct body, and in the NGINX logs I can see the following:
172.23.0.1 - - [02/Feb/2023:20:20:10 +0000] "GET / HTTP/1.1" 302 5 "-" "lwp-request/6.61 libwww-perl/6.61" "-" 0.086 0.088 . -
172.23.0.1 - - [02/Feb/2023:20:20:10 +0000] "GET /index.php?owa_do=base.loginForm&owa_go=http%3A%2F%2F127.0.0.1%3A8080%2F& HTTP/1.1" 200 3200 "-" "lwp-request/6.61 libwww-perl/6.61" "-" 0.086 0.088 . -
Doing it in Python with the following basic code also works and gives similar results as with the terminal GET version:
import requests
x = requests.get("http://127.0.0.1:8080/")
print(x.content)
What am I doing wrong?

Got it working with following redirects (see here):
begin
response = Net::HTTP.get_response(URI.parse(url))
url = response['location']
end while response.is_a?(Net::HTTPRedirection)

Why does this API print a log on as output when I make a get request?

I am using PyCharm and when I make a get request, I get this message in red text:
30-Jun-19 11:32:21 DEBUG Starting new HTTPS connection (1): eodhistoricaldata.com:443
30-Jun-19 11:32:21 DEBUG https://eodhistoricaldata.com:443 "GET /api/real-time/AAL.US?api_token=MYAPIKEY&fmt=json HTTP/1.1" 200 None
Another example is :
30-Jun-19 11:32:12 INFO POST Request: https://api.twilio.com/2010-04-01/Accounts/MYACCOUNTINFO/Messages.json
Other APIs I use will make calls silently and others will display this red text(INFO, DEBUG, etc)
So what is this red text and how can I have it not printing to my console?
Thanks

I found an answer so I will post for future people to see:
import logging
logging.disable()

Extract data from Get requests in python

I have got a SimpleHTTPServer in a raspberry pi working with a python script that is executed in the same directory as the index.html web page. The code is the following:
#!/usr/bin/python
import SimpleHTTPServer
import SocketServer
from BaseHTTPServer import BaseHTTPRequestHandler, HTTPServer
import httplib2
PORT = 8080
Handler = SimpleHTTPServer.SimpleHTTPRequestHandler
httpd = SocketServer.TCPServer(("", PORT), Handler)
print "serving at port", PORT
httpd.serve_forever()
Once the web page is loaded, it starts to send different get requests with data that I need to read with my python script, but I don know how to do it.
This is an example of the get requests;
10.8.0.6 - - [27/Nov/2016 11:18:07] code 404, message File not found
10.8.0.6 - - [27/Nov/2016 11:18:07] "GET /ok.png HTTP/1.1" 404 -
10.8.0.6 - - [27/Nov/2016 11:18:07] code 404, message File not found
10.8.0.6 - - [27/Nov/2016 11:18:07] "GET /red.png HTTP/1.1" 404 -
10.8.0.6 - - [27/Nov/2016 11:18:07] code 404, message File not found
10.8.0.6 - - [27/Nov/2016 11:18:07] "GET /arduino/start/0.16388046142178503 HTTP/1.1" 404 -
10.8.0.6 - - [27/Nov/2016 11:18:07] code 404, message File not found
10.8.0.6 - - [27/Nov/2016 11:18:07] "GET /favicon.ico HTTP/1.1" 404 -
10.8.0.6 - - [27/Nov/2016 11:18:17] code 404, message File not found
10.8.0.6 - - [27/Nov/2016 11:18:17] "GET /arduino/update/0.6913944096802204 HTTP/1.1" 404 -
Don't worry about the 404 error, it appears because the web has different icons that I did not put in the directory yet because I am doing tests.
The reason because I want to do that it's because I am "translating" a web sever to control a solenoid valve and different sensors that I had working on Arduino Yun. In arduino, the code reads the get requests, and depending on what is the get request asking for it responses with an XML package with the sensor data or it acts on the valve.
On arduino, the code for "catching" the get request as a String is the following;
if (client) { // got client?
String request = client.readString();
request.trim();
And then the program looks for a word in the string and depending on what word is found it responses with different information. An example;
if (request=="inicial") {
// send rest of HTTP header
client.println("Content-Type: text/xml");
client.println("Connection: keep-alive");
client.println();
// send XML file containing input states
inicial(client);
}
Summaraizing, I want to read the get requests as a String in my python script and extract data from them, and then response with an XML, but I think this last step will be better explained on the internet so I don't worry about it.
I hope my explanation was enough clear.
Thanks!!!

This is how you can make request to an address. The response object is assigned to the variable. Then you can extract anything you want from it.
>>> r = requests.get('https://api.github.com/user', auth=('user', 'pass'))
>>> r.status_code
200
>>> r.headers['content-type']
'application/json; charset=utf8'
>>> r.encoding
'utf-8'
>>> r.text
u'{"type":"User"...'
>>> r.json()
{u'private_gists': 419, u'total_private_repos': 77, ...}

AppEngine serves blank page on the first request - what is the difference between first and subsequent instance loads?

I am having a problem with the AppEngine serving me a blank page every time the instance is loaded.
It is similar problem already covered here, however, there was no constructive solution mentioned that would help my case.
As there are no errors in the console, this one seems to be extremely hard to debug and I frankly do not know how to start. I have not solution mentioned in the previous post as I do not want to rename any of my files and I want to keep my code organised as it is now.
Lastly, to ask a broader question - what is the difference the first and subsequent requests are handled for each of the instances? Is there anything a developer should be aware of?
So what exactly happens - if the appengine's code is loaded for the first time, I get a blank page. Every subsequent request is fine. I think I have all the required parts of the code there, just in case here are the handlers from app.yaml (also, the app is using python27 as a platform):
handlers:
- url: /favicon\.ico
static_files: favicon.ico
upload: favicon\.ico
- url: /fb/.*
script: fbhandler.py
- url: /xarpc/.*
script: xmlrpchandler.py
- url: /
script: main.py
- url: .*
script: main.py
The handler in question is the fbhandler.py.
def main():
application = webapp.WSGIApplication([("/fb/", FBHandler)], debug=True)
util.run_wsgi_app(application)
if __name__ == '__main__':
main()
If I examine the appengine logs, I see this for the "blank" request:
2012-05-07 10:43:14.822 /fb/?id=341108955956205 200 2998ms 0kb Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.168 Safari/535.19
95.23.245.xx - - [07/May/2012:03:43:14 -0700] "POST /fb/?id=xxx HTTP/1.1" 200 0 - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.168 Safari/535.19" "xxx.appspot.com" ms=2999 cpu_ms=563 api_cpu_ms=0 cpm_usd=0.015744 loading_request=1 instance=00c61b117cb0ca2fc61adc3939c4bd034dfa416f
I 2012-05-07 10:43:14.822
This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application.
Compared to this for any good subsequent request:
2012-05-07 10:44:05.479 /fb/?id=341108955956205 200 832ms 0kb Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.168 Safari/535.19
95.23.245.xx - - [07/May/2012:03:44:05 -0700] "POST /fb/?id=xxx HTTP/1.1" 200 483 - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.168 Safari/535.19" "xxx.appspot.com" ms=832 cpu_ms=1272 api_cpu_ms=1078 cpm_usd=0.035497 instance=00c61b117cb0ca2fc61adc3939c4bd034dfa416f
As you can see, no obvious error messages anywhere.
Update:
Upon advice, I have removed the main() to try to rule out caching, now the handler looks like:
application = webapp.WSGIApplication([("/fb/", FBHandler)], debug=True)
util.run_wsgi_app(application)
Same effect, first load gets blank page and subsequent loads get proper data. What I have noticed that it does not matter if a new code is uploaded or not, if an instance is running and I update the code, it shows properly on the first load. I need to go into instances admin and shut one down to reproduce the error.
Update 2:
I have now tried changing the handlers to the new python27 way:
- url: /fb/.*
script: fbhandler.app
And also corresponding change to the fbhandler
import webapp2
app = webapp2.WSGIApplication([("/fb/", FBHandler)])
Again, same issue. I have also tried removing the debug parameter with no effect.

You are experiencing this behavior due to the way App Engine caches your CGI handler.
If you look in the runtime documentation (CGI Handler Scripts Can Also Be Cached
section) you can see that:
You can tell App Engine to cache the CGI handler script itself, in addition to imported modules. If the handler script defines a function named main(), then the script and its global environment will be cached like an imported module. The first request for the script on a given web server evaluates the script normally. For subsequent requests, App Engine calls the main() function in the cached environment.
To be sure that your handler works properly even during a instance startup be sure to define your WSGI application as a global variable (outside of main).

SOLUTION
I now seem to have found a solution. When I upgraded all of the application handlers to the webapp2 and used the new way of referencing them, it now seems to work without the first load error.

Blank Page But No Error - Python Appengine

I'm receiving intermittent blank pages on my appengine python website. Typically these come when a new process is started or when I flush the cache. There is a single white page served and once that has served everything is fine.
It's basically the same error as here:
http://groups.google.com/group/google-appengine/browse_thread/thread/c072383dc970e450
However, I have double and triple checked that I have the correct code on my python file (the following is copied and pasted):
def main():
run_wsgi_app(application)
if __name__ == "__main__":
main()
Here is an example response in the logs that generated the blank page:
01-02 04:46AM 48.539 / 200 188ms 570cpu_ms 383api_cpu_ms 0kb
Mozilla/5.0 (Windows; U; Windows NT
6.0; en-US) AppleWebKit/534.13 (KHTML, like Gecko) Chrome/9.0.597.19
Safari/534.13,gzip(gfe),gzip(gfe),gzip(gfe)
86.164.42.252 - tjcritchlow [02/Jan/2011:04:46:48 -0800] "GET /
HTTP/1.1" 200 124 - "Mozilla/5.0
(Windows; U; Windows NT 6.0; en-US)
AppleWebKit/534.13 (KHTML, like Gecko)
Chrome/9.0.597.19
Safari/534.13,gzip(gfe),gzip(gfe),gzip(gfe)"
"www.7bks.com" ms=188 cpu_ms=570
api_cpu_ms=383 cpm_usd=0.016028
I 01-02 04:46AM 48.724
Saved; key: appstats:008500, part: 82 bytes, full: 92081 bytes,
overhead: 0.001 + 0.005; link:
http://www.7bks.com/stats/details?time=1293972408543
Any suggestions welcome on how I might debug further or solve this issue.
I have a couple of different python files, here's the handlers from my app.yaml. But I've checked all of them to ensure they all have the correct if name code at the bottom.
handlers:
- url: /admin/.*
script: admin.py
login: admin
- url: /googleanalytics/
script: googleanalytics.py
login: admin
- url: /cleanupsessions/
script: cleanupsessions.py
login: admin
- url: /robots.txt
static_files: robots.txt
upload: robots.txt
- url: /favicon.ico
static_files: images/favicon.ico
upload: images/favicon.ico
- url: /images
static_dir: images
- url: /css
static_dir: css
- url: /jquery
static_dir: jquery
- url: /.*
script: 7books.py
error_handlers:
- file: customerror.html
Could the issue be with one of the libraries I'm importing? Should I check all of them to ensure they all have the name code?

Regarding the blank page issue, debugging that might be a bit involved unless you can narrow down the problem to a particular piece of your code or the GAE stack.
Appstats is a tool for GAE that hooks into each web request (as WSGI middleware) and records performance, debugging, and some other information from requests, aggregating them in an administrative interface which you can access through your GAE admin site. It's a nice way of seeing tracebacks for errors, and monitoring your site for errors (which show up as yellow boxed "E" line items in the request listing once you've installed appstats).
First, you should set up Appstats by following the directions at:
http://code.google.com/appengine/docs/python/tools/appstats.html
...which will tell you loads of useful information about every
request, especially ones with errors, including a Python traceback
containing what went wrong and where in the call stack, just like you're used to when debugging errors with your code locally using the Google App Engine Launcher or even just Python CLI or iPython.
Then, next time you see a blank page, hop over to your admin page or
to appstats, log in, and you'll have a traceback that will tell you
where in your code something broke.
Most likely, this is just something returning prematurely or an
unhandled edge case somewhere in your code, but it could just as
easily be cosmic rays until you have appstats to look at your request
logs and debug the error.
Once you've installed Appstats and reproduced the problem, paste the traceback above, (remember to remove any user information such as passwords that may be in the traceback), and I'll revise this response to match.

i just resolved this after lots of frustration.
i had renamed my script handler to 'site.py', renaming it back to 'main.py' resolved everything.
My best guess is that the stock python 'site' module was dominant on the classpath over my site.py. if someone knows more, please leave a comment here.

Make sure your python indentation is consistent. For instance, either use tabs or 4 spaces or 2 spaces... don't mix them within the same function. This solved it for me! Which is annoying since a tab and 4 spaces look exactly the same to a human.