This question already has answers here:
Ruby - net/http - following redirects
(6 answers)
Closed 17 days ago.
I'm currently simply trying to get a simple GET request working in Ruby, however, I'm seeing some strange behavior.
I have an Open Web Analytics application running with Docker and it is reachable at http://127.0.0.1:8080/.
I can reach the login site and everything works fine.
Now I want to do a GET request with Ruby to analyze the body of that request but I cannot get it to work, in other languages like Python or simple GET requests over the terminal it works fine. Why not with Ruby?
Here is my very basic Ruby code:
require 'net/http'
url = 'http://127.0.0.1:8080/'
uri = URI(url)
session = Net::HTTP.new(uri.host, uri.port)
response = session.get(uri.request_uri)
puts response.body
Which doesn't output anything. If I look into the NGINX logs from the container, I can see the request being made but there is no further redirection as with the other methods (see below).
172.23.0.1 - - [02/Feb/2023:20:02:59 +0000] "GET / HTTP/1.1" 302 5 "-" "Ruby" "-" 0.088 0.088 . -
If I do a simple GET over the terminal, it works:
GET http://127.0.0.1:8080/
will output the correct body, and in the NGINX logs I can see the following:
172.23.0.1 - - [02/Feb/2023:20:20:10 +0000] "GET / HTTP/1.1" 302 5 "-" "lwp-request/6.61 libwww-perl/6.61" "-" 0.086 0.088 . -
172.23.0.1 - - [02/Feb/2023:20:20:10 +0000] "GET /index.php?owa_do=base.loginForm&owa_go=http%3A%2F%2F127.0.0.1%3A8080%2F& HTTP/1.1" 200 3200 "-" "lwp-request/6.61 libwww-perl/6.61" "-" 0.086 0.088 . -
Doing it in Python with the following basic code also works and gives similar results as with the terminal GET version:
import requests
x = requests.get("http://127.0.0.1:8080/")
print(x.content)
What am I doing wrong?
Got it working with following redirects (see here):
begin
response = Net::HTTP.get_response(URI.parse(url))
url = response['location']
end while response.is_a?(Net::HTTPRedirection)
Related
I tried a bunch of tutorials, here is an example of one of them
https://nickmccullum.com/build-facebook-bot-python-flask/
Request is send:
127.0.0.1 - - [02/May/2022 10:18:15] "GET /?hub.mode=subscribe&hub.challenge=646416116&hub.verify_token=YOUR_VERIFY_TOKEN HTTP/1.1" 200 -
But when trying to validate, the Webhook throws this error:
Validation of the callback URL or confirmation token failed. Confirm the information provided or try again later.
what could be the problem
So what I am trying to do is to extract all urls from HTTP requests list. They should be stripped of protocol, parameters and slash at the end of the path(if exists).So for example:
10.4.180.222 [5/Feb/2018:08:03:40 +0100] "GET http://somewebsite.com/ HTTP/1.1" 200 1080
10.4.180.222 [5/Feb/2018:08:03:11 +0100] "GET http://www.somewebsite.cc/somesubdomain/ HTTP/1.1" 200 3056
10.4.180.222 [5/Feb/2018:08:03:11 +0100] "GET https://www.somewebsite.ua HTTP/1.1" 200 3056
Should be:
somewebsite.com
www.somewebsite.cc/somepath
www.somewebsite.ua
I've tried to do this in two steps, without using any sophisticated regex(just general for any url)
urls = re.findall('http[s]?://(?:[a-zA-Z]|[0-9]|[$-_#.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+', file.read())
And then using urlparse.
domain = '{url.netloc}{url.path}'.format(url=urlparse(url))
It works almost fine. However I am getting path ending with slash.
www.somewebsite.cc/somepath/
So I've decided to use regex. However, I know only basics so I can't come up with anything well-functioning.Right now I have something like that but it doesn't cover "/" thing and different protocols :/
Thank you for any advice :)
((?:www\.+)[A-Za-z0-9\.\-]+)((?:\/[\+~%\/\.\w\-]*))
If the end slash is your only problem, this is the solution.
urls = [ x.rstrip('/') for x in re.findall('http[s]?://(?:[a-zA-Z]|[0-9]|[$-_#.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+', file.read()) ]
In other words, just do
urls = [ x.rstrip('/') for x in < your regex goes here > ].
I use python and google-api-python-client==1.6.2 to work with user Google Drive's. Everything works perfectly without any bugs, but I didn't receive any push notification from Google Drive (Google Drive API Documentation). "Registering your domain" successfully completed. "Creating notification channels" completed with Channel object returning to me.
My code:
try:
drive_service.files().watch(
fileId='CPw3cbyqkoC1QMK48R24-Z2CG9w',
body=dict(
id=str(uuid.uuid4()),
resourceId='CPw3cbyqkoC1QMK48R24-Z2CG9w',
type='web_hook',
address='https://my-domain-address/web_hook'
)
).execute()
except HttpError as err:
logger.exception('HttpError {}: content={}'.format(err.uri, err.content))
Response:
{
u'resourceId': u'CPw3cbyqkoC1QMK48R24-Z2CG9w',
u'kind': u'api#channel',
u'expiration': u'1495448262000',
u'id': u'8837a4ad-98c0-4e89-8899-c07e12e3bffc',
u'resourceUri': u'https://www.googleapis.com/drive/v3/files/0B2lHB_g_GJY9RWx6UkRjWUFjSVU?acknowledgeAbuse=false&alt=json&supportsTeamDrives=false&alt=json'
}
After it there are not any requests to https://my-domain-address/web_hook when User modify watched Google Drive resource.
Have you any ideas what I do wrong?
Log messages linked with the problem that I found while search a solution:
64.233.172.141 - - [22/May/2017:09:11:10 +0000] "POST /web_hook HTTP/1.1" 403 807 "-" "APIs-Google; (+https://developers.google.com/webmasters/APIs-Google.html)"
64.233.172.143 - - [22/May/2017:09:11:12 +0000] "POST /web_hook HTTP/1.1" 403 807 "-" "APIs-Google; (+https://developers.google.com/webmasters/APIs-Google.html)"
64.233.172.143 - - [22/May/2017:09:11:13 +0000] "POST /web_hook HTTP/1.1" 403 807 "-" "APIs-Google; (+https://developers.google.com/webmasters/APIs-Google.html)"
So I'm receiving the Google Drive requests to my webhook.
Firstly I think that Nginx is blocking them, but after I had seen that all requests are POST.
So I came to the problem of forwarding POST requests. Also I use Django and forgot disable CSRF protection to webhook. This is a solution of my problem.
I'll be glad if it helps someone.
I have got a SimpleHTTPServer in a raspberry pi working with a python script that is executed in the same directory as the index.html web page. The code is the following:
#!/usr/bin/python
import SimpleHTTPServer
import SocketServer
from BaseHTTPServer import BaseHTTPRequestHandler, HTTPServer
import httplib2
PORT = 8080
Handler = SimpleHTTPServer.SimpleHTTPRequestHandler
httpd = SocketServer.TCPServer(("", PORT), Handler)
print "serving at port", PORT
httpd.serve_forever()
Once the web page is loaded, it starts to send different get requests with data that I need to read with my python script, but I don know how to do it.
This is an example of the get requests;
10.8.0.6 - - [27/Nov/2016 11:18:07] code 404, message File not found
10.8.0.6 - - [27/Nov/2016 11:18:07] "GET /ok.png HTTP/1.1" 404 -
10.8.0.6 - - [27/Nov/2016 11:18:07] code 404, message File not found
10.8.0.6 - - [27/Nov/2016 11:18:07] "GET /red.png HTTP/1.1" 404 -
10.8.0.6 - - [27/Nov/2016 11:18:07] code 404, message File not found
10.8.0.6 - - [27/Nov/2016 11:18:07] "GET /arduino/start/0.16388046142178503 HTTP/1.1" 404 -
10.8.0.6 - - [27/Nov/2016 11:18:07] code 404, message File not found
10.8.0.6 - - [27/Nov/2016 11:18:07] "GET /favicon.ico HTTP/1.1" 404 -
10.8.0.6 - - [27/Nov/2016 11:18:17] code 404, message File not found
10.8.0.6 - - [27/Nov/2016 11:18:17] "GET /arduino/update/0.6913944096802204 HTTP/1.1" 404 -
Don't worry about the 404 error, it appears because the web has different icons that I did not put in the directory yet because I am doing tests.
The reason because I want to do that it's because I am "translating" a web sever to control a solenoid valve and different sensors that I had working on Arduino Yun. In arduino, the code reads the get requests, and depending on what is the get request asking for it responses with an XML package with the sensor data or it acts on the valve.
On arduino, the code for "catching" the get request as a String is the following;
if (client) { // got client?
String request = client.readString();
request.trim();
And then the program looks for a word in the string and depending on what word is found it responses with different information. An example;
if (request=="inicial") {
// send rest of HTTP header
client.println("Content-Type: text/xml");
client.println("Connection: keep-alive");
client.println();
// send XML file containing input states
inicial(client);
}
Summaraizing, I want to read the get requests as a String in my python script and extract data from them, and then response with an XML, but I think this last step will be better explained on the internet so I don't worry about it.
I hope my explanation was enough clear.
Thanks!!!
This is how you can make request to an address. The response object is assigned to the variable. Then you can extract anything you want from it.
>>> r = requests.get('https://api.github.com/user', auth=('user', 'pass'))
>>> r.status_code
200
>>> r.headers['content-type']
'application/json; charset=utf8'
>>> r.encoding
'utf-8'
>>> r.text
u'{"type":"User"...'
>>> r.json()
{u'private_gists': 419, u'total_private_repos': 77, ...}
Trying to learn flask but stuck with some error or maybe an issue.
def check_int(no):
return "number is %d" %no
app.add_url_rule('/hello/<int:no>', 'nothign_specific', check_int)
So when I do a curl call to http://127.0.0.1:5000/hello/1 it fails wherein the same curl call to any other number apart from 1 passes.
http://127.0.0.1:5000/hello/<any number apart from 1 passes>
127.0.0.1 - - [05/Aug/2016 14:17:48] "GET /hello/1/ HTTP/1.1" 404 -
127.0.0.1 - - [05/Aug/2016 14:18:01] "GET /hello/12 HTTP/1.1" 200 -
Can someone let me know what's happening around
In flask, if your route (or rule) definition has no trailing slash is explicit. If you would add a trailing / to your url rule, i.e.
'/hello/<int:no>/'
then you would be able to use both (request with or without /).
According to flask docs, a route with a trailing slash is treated similar to a folder name in a file system: If accessed without the slash, flask will recognize it and redirect you to the one with slash. Contrastingly, a route that is defined without a trailing slash is treated like the pathname of a file, i.e. it will throw 404 when accessed with a trailing slash.
Read more: http://flask.pocoo.org/docs/0.11/quickstart/, section "Unique URLs / Redirection Behavior"