I am a beginner so I apologize if my question is very obvious or not worded correctly.
I need to send a request to a URL so data can then be sent back in XLM format. The URL will have a user specific login and password, so I need to incorporate that as well. Also there is a port (port 80) that I need to include in the request. Is requests.get the way to go? I'm not exactly sure where to start. After receiving the XLM data, I need to process it (store it) on my machine - if anyone also wants to take a stab at that (I am also struggling to understand exactly how XLM data is sent over, is it an entire file?). Thanks in advance for the help.
Here is a python documentation on how to fetch internet resources using the urllib package.
It talks about getting the data, storing it in a file, sending data and some basic authentication.
https://docs.python.org/3/howto/urllib2.html
Getting the URL would look something like this import
Import urllib.request
urllib.request.urlopen("http://yoururlhere.co.uk").read()
Note that this is for strings and Python 3 only.
Python 2 version can be found here
What is the quickest way to HTTP GET in Python?
If you want to parse the data you may want to use this
https://docs.python.org/2/library/xml.etree.elementtree.html
I hope this helps! I am not too sure on how you would approach the username and password stuff but these links can hopefully provide you with information on how to do some of the other stuff!
Import the requests library and then call the post method as follows:
import requests
data = {
"email" : "netsparkertest#test.com",
"password" : "abcd12333",
}
r = requests.post('www.facebook.com', data=data)
print r.text
print r.status_code
print r.content
print r.headers
Related
How can I know if a website is using apache, nginx or other and get this information in python? Thanks in advance
This information if available is given in the header of the response to a HTTP Request. With Python you can perform HTTP requests using the module requests.
Make a simple GET request to the interested site and then print the headers parameter of the returned object.
import requests
r = requests.get(YOUR_SITE)
print(r.headers)
The output is made of a dictionary of keys and value, you have to look for the Server parameter
server = r.headers['Server']
You must be aware that not all websites return this information for several reasons, so you could not find this key in the response header.
I am having some trouble running the Freedom of information act API in python. I am sure it is related to how I am implementing my API key but I am uncertain as to where I am dropping the ball. Any help is greatly appreciated.
import requests
apikey= ''
api_base_url = f"https://api.foia.gov/api/webform/submit"
endpoint = f"{api_base_url}{apikey}"
r = requests.get(endpoint)
print(r.status_code)
print(r.text)
there error I receive is requests.exceptions.InvalidSchema: No connection adapters were found for this website. Thanks again
According to the documentation, the API requires the API key to be passed as a request header parameter ("X-API-Key"). Your python code appears to be simply concatenating the API key and the URL.
The following Q&A explains how to set a request header using requests.
Using headers with the Python requests library's get method
It would be something like this:
import requests
apikey= ...
api_base_url = ...
r = requests.get(api_base_url,
headers={"X-API-Key": apikey})
print(r.status_code)
print(r.text)
Note that the documentation for the FOIA site explains what you need to do to submit a FIOA request form. It is significantly different to what your Python code is apparently trying to do. I would advise you to read the documentation. Also read the manual entry for the "curl" command so that you understand the requests that the examples show.
I took a programming class in python so I know the basics of the language. A project I'm currently attempting involves submiting a form repeatedly untill the request is successful. In order to achieve a faster success using the program, I thought cutting the browser out of the program by directly sending and recieving data from the server would be faster. Also the website I'm creating the program for has a tendency to crash but I'm pretty sure i could still receive and send response to the server. Currently, im just researching different resources I could use to complete the task. I understand mechanize is easy to fill forms and submit them, but it requires a browser. So my question is what would be the best resource to use within python to communicate directly with the server without a browser.
I apologize if any of my knowledge is flawed. I did take the class but I'm still relatively new to the language.
Yes, there are plenty of ways to do this, but the easiest is the third party library called requests.
With that installed, you can do for example:
requests.post("https://mywebsite/path", {"key: "value"})
You can try this below.
from urllib.parse import urlencode
from urllib.request import Request, urlopen
url = 'https://httpbin.org/post' # Set destination URL here
post_fields = {'foo': 'bar'} # Set POST fields here
request = Request(url, urlencode(post_fields).encode())
json = urlopen(request).read().decode()
print(json)
I see from your tags that you've already decided to use requests.
Here's how to perform a basic POST request using requests:
Typically, you want to send some form-encoded data — much like an HTML
form. To do this, simply pass a dictionary to the data argument. Your
dictionary of data will automatically be form-encoded when the request
is made
import requests
payload = {'key1': 'value1', 'key2': 'value2'}
response = requests.post("http://httpbin.org/post", data=payload)
print(response.text)
I took this example from requests official documentation
I suggest you to read it and try also other examples available in order to become more confident and decide what approach suits your task best.
I am a beginner trying to learn REST API programming through Python 2.7 to get data from Socialcast API. From my research it looks like requests or urllib2 would work. I need to authenticate with username and id for the API. I tried using urllib2 and it gave me error 401.
Which one should I use? My goal is to produce .csv files from the data so I can visualize it. Thank you in advance.
The question will yield a bit of an opinion based response, but I would suggest using Requests. I find that when making request that require parameters using Requests is easier to manage. An example for the Socialcast using Requests would be
parameters={"email" : emailAddress, "passoword" : password}
r = requests.post(postUrl, parameters)
The post url would be the url to make the post request and emailAddress and password would be the vales you use to login in.
For the csv, take a look here which includes a tutorial on going from json to csv.
I'm coding an app which has to use this api. So I want to do at a certain point a search on their database. Now I'm struggling with which python library is the right one to use in order to authenticate about oAuth2? I couldn't find any by now, where I was sure, it would offer the necessary functions.
I wonder if this library (python-oauth2) offers, what I need. But this isn't a library for the client, is it? It seems it is for the server...
I'd be really grateful, if someone could just give me an advice, with what I should work.
Method 1
You will need to use the following modules. No need to use oauth. Just need to get the token before performing any search using the api.
requests, json, urllib
Here's a short Example code for that
import requests, json, urllib
BASE_URL = "http://scoilnet.com/grants/apikey/"
r = requests.post(_BASE_URL+"user/token/", data={'username': username , 'password': password })
print r.getcontent
The above code will show you how to request a token from the api. Using that token you will be making get and post requests to the api which will give a json response. That json response will be shown as a Dictionary from which you will load you data in your program.
Method 2
You can also use urllib or urllib2 or urllib3