Hi I am very new to python programming. Here I'm trying to write a python script which will get a status code using GET request. I can able to do it for single URL but how to do it for multiple URL's in a single script.
Here is the basic code I have written which will get response code from a url.
import requests
import json
import jsonpath
#API URL
url = "https://reqres.in/api/users?page=2"
#Send Get Request
response = requests.get(url)
if response:
print('Response OK')
else:
print('Response Failed')
# Display Response Content
print(response.content)
print(response.headers)
#Parse response to json format
json_response = json.loads(response.text)
print(json_response)
#Fetch value using Json Path
pages = jsonpath.jsonpath(json_response,'total_pages')
print(pages[0])
try this code.
import requests
with open("list_urls.txt") as f:
for url in f:
response = requests.get(url)
print ("The url is ",url,"and status code is",response.status_code)
I hope this helps.
You can acess to the status code with response.status_code
You can put your code in a function like this
def treat_url(url):
response = requests.get(url)
if response:
print('Response OK')
else:
print('Response Failed')
# Display Response Content
print(response.content)
print(response.headers)
#Parse response to json format
json_response = json.loads(response.text)
print(json_response)
#Fetch value using Json Path
pages = jsonpath.jsonpath(json_response,'total_pages')
print(pages[0])
And have a list of urls and iterate throw it:
url_list=["www.google.com","https://reqres.in/api/users?page=2"]
for url in url_list:
treat_url(url)
A couple of suggestions, the question itself is not very clear, so a good articulation would be useful for all the contributors over here :) ...
Now coming to what I was able to comprehend, there are few modifications that you can do:
response = requests.get(url) You will always get a response object, I think you might want to check the status code here, which you can do by response.status_code and based upon what you get, you say whether or not you got a success response.
and regarding looping, you can check the last page from response JSON as response_json['last_page'] and run a for loop on range(2, last_page + 1) and append the page number in URI to fetch individual pages response
You can directly fetch JSON from response object response.json()
Please refer to requests doc here
Related
I'm trying to get my head around the requests python package
import requests
url = "https://www.google.com/search?q=london"
response = requests.get(url, headers={"Accept": "application/json"})
data = response.json()
And i'm receiving the following error:
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
However this code does work with some other websites.. is there a reason this would error on specific websites and is there a way around it? For example if i wanted the search results when searching London on Google?
Thanks
response.json() does not convert any server response to a JSON, it simply parses 'stringified' JSONs. So if the server returns a string that is not a JSON, then this will throw a decode error.
Some servers do return JSON objects, in which case your code will work. In the case of https://www.google.com/search?q=london this actually returns HTML code (as you would expect since it's a webpage).
You can test this by printing the response:
print(response.text)
which outputs:
# some very long output that ends with:
...();})();google.drty&&google.drty();</script></body></html>
Notice the </html> tag at the end? So this cannot be parsed into a JSON.
So how do you parse this into a usable HTML? You can use beautiful soup:
import requests
from bs4 import BeautifulSoup
url = "https://www.google.com/search?q=london"
response = requests.get(url, headers={"Accept": "application/json"})
soup = BeautifulSoup(response.text)
print(soup.prettify())
I am new to external scripting in Service Now.
I have this requirement of performing a text search in Service Now. I'm planning to insert the texts to be searched in a csv file and then through a python rest API of service now, perform the text search.
However, I was unable to find any direct python REST Service Now API to do text search. Does anyone know any alternative ways of doing this task?
Thanks in Advance!
You can use 'Global Text Search API' in ServiceNow platform to get search results. First you need to get the list of search groups available and then send a query with list of groups that need to be searched and the search keyword.
Here's a sample program that searched for keyword agent in Knowledge base and Service catalog.
import requests
# Set the request parameters
url = 'https://<instance>.service-now.com/api/now/v1/globalsearch/groups'
# Eg. User name="admin", Password="admin" for this code sample.
user = 'admin'
pwd = 'admin'
# Set proper headers
headers = {"Content-Type":"application/json","Accept":"application/json"}
# Do the HTTP request
response = requests.get(url, auth=(user, pwd), headers=headers )
# Check for HTTP codes other than 200
if response.status_code != 200:
print('Status:', response.status_code, 'Headers:', response.headers, 'Error Response:',response.json())
exit()
# Decode the JSON response into a dictionary and use the data
data = response.json()
print(data)
url = 'https://<instance>.service-now.com/api/now/v1/globalsearch/search?sysparm_groups=2b77ce594bd6c0d901b8fbe25ceff671&sysparm_search=agent'
# Do the HTTP request
response = requests.get(url, auth=(user, pwd), headers=headers )
# Check for HTTP codes other than 200
if response.status_code != 200:
print('Status:', response.status_code, 'Headers:', response.headers, 'Error Response:',response.json())
exit()
# Decode the JSON response into a dictionary and use the data
data = response.json()
print(data)
Greetings Team Good Day,
I am using the urlib code to print the response data but getting the HTTP error, I have attached the error screenshot and the code in the post, request your advise.
enter image description here
import urllib.parse
import urllib.request
url = 'https://www.google.com/search'
values = {'q': 'python programming tutorials'}
data = urllib.parse.urlencode(values)
data = data.encode('utf-8')
request = urllib.request.Request(url, data)
response = urllib.request.urlopen(request)
HTTPError: HTTP Error 405: Method Not Allowed
You're using a POST method, which is not allowed in this url:
>> request.get_method()
'POST'
Google search expects a GET.
Every time you include the data in the request, it will automatically be a POST. But instead, you can do this directly:
response = urllib.request.urlopen('{0}?{1}'.format(url, data))
I'm relatively new to Python so would like some help, I've created a script which simply use the request library and basic auth to connect to an API and returns the xml or Json result.
# Imports
import requests
from requests.auth import HTTPBasicAuth
# Set variables
url = "api"
apiuser = 'test'
apipass = 'testpass'
# CALL API
r = requests.get(url, auth=HTTPBasicAuth(apiuser, apipass))
# Print Statuscode
print(r.status_code)
# Print XML
xmlString = str(r.text)
print(xmlString)
if but it returns a blank string.
If I was to use a browser to call the api and enter the cretentials I get the following response.
<Response>
<status>SUCCESS</status>
<callId>99999903219032190321</callId>
<result xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="Dummy">
<authorFullName>jack jones</authorFullName>
<authorOrderNumber>1</authorOrderNumber>
</result>
</Response>
Can anyone tell me where I'm going wrong.
What API are you connecting to?
Try adding a user-agent to the header:
r = requests.get(url, auth=HTTPBasicAuth(apiuser, apipass), headers={'User-Agent':'test'})
Although this is not an exact answer for the OP, it may solve the issue for someone having a blank response from python-requests.
I was getting a blank response because of the wrong content type. I was expecting an HTML rather than a JSON or a login success. The correct content-type for me was application/x-www-form-urlencoded.
Essentially I had to do the following to make my script work.
data = 'arcDate=2021/01/05'
headers = {
'Content-Type': 'application/x-www-form-urlencoded',
}
r = requests.post('https://www.deccanherald.com/getarchive', data=data, headers=headers)
print(r.status_code)
print(r.text)
Learn more about this in application/x-www-form-urlencoded or multipart/form-data?
Run this and see what responses you get.
import requests
url = "https://google.com"
r = requests.get(url)
print(r.status_code)
print(r.json)
print(r.text)
When you start having to pass things in your GET, PUT, DELETE, OR POST requests, you will add it in the request.
url = "https://google.com"
headers = {'api key': 'blah92382377432432')
r = requests.get(url, headers=headers)
Then you should see the same type of responses. Long story short,
Print(r.text) to see the response, then you once you see the format of the response you get, you can move it around however you want.
I have an empty response only when the authentication failed or is denied.
The HTTP status is still ≤ 400.
However, in the header you can find :
'X-Seraph-LoginReason': 'AUTHENTICATED_FAILED'
or
'X-Seraph-LoginReason': 'AUTHENTICATED_DENIED'
If the request is empty, not even a status code I could suggest waiting some time between printing. Maybe the server is taking time to return the response to you.
import time
time.sleep(5)
Not the nicest thing, but it's worth trying
How can I make a time delay in Python?
I guess there are no errors during execution
EDIT: nvm, you mentioned that you got a status code, I thought you were literally geting nothing.
On the side, if you are using python3 you have to use Print(), it replaced Print
I want to use urllib2 through a proxy http site, post to the form component having name="what", click submit, and return the resulting webpage as a string. I know many have asked this question before, see here for example. However, I couldn't get their solutions to work for my example code below:
url = "http://anonymouse.org/anonwww.html"
posturl = "www.google.ca"
values = {'what':posturl}
data = urllib.urlencode(values)
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
html = response.read()
print html
piggybacking on Christian's answer:
requests is a very good library for this stuff... however, urllib2 also suffices:
import urllib2
def get_anon_content(url):
anon_url = 'http://anonymouse.org/cgi-bin/anon-www.cgi/%s' % url
req = urllib2.Request(anon_url)
response = urllib2.urlopen(req)
content = response.read()
return content
url = 'http://www.google.ca'
print get_anon_content(url)
in youre case you can just use this url:
http://anonymouse.org/cgi-bin/anon-www.cgi/http://www.google.ca
its the same thing as using anonymouse except you dont have to go to the site you just use the url
and next time make it easier on youreself and use requests you can get the same effect of urllib but with like 4 lines so check that out
and good luck :)