Develop Reference

Python Download Roblox Data

I'm attempting to format json from the Roblox API. I have a names.txt that stores all of the names. This is how the file looks
rip_robson0007
Abobausrip
app_58230
kakoytochelik123
Ameliathebest727
Sherri0708
HixPlayk
mekayla_091
ddddorffg
ghfgrgt7nfdbfj
TheWolfylikedog
paquita12345jeje
hfsgfhsgfhgfhds
It stores a bunch of usernames seperated by a new line. The code is suppose to use the names and for each name get the JSON from this endpoint https://api.roblox.com/users/get-by-username?username={name} & format it as I have in my code. It always returns error 429 and doesn't save any of the data.
This is the code:
import json
import requests
import time
# Read the names from the text file
with open("./txt/names.txt", "r") as f:
names = f.read().split("\n")
# Initialize an empty dictionary to store the users
users = {}
# Iterate through the names
for name in names:
time.sleep(5)
response = requests.get(f"https://api.roblox.com/users/get-by-username?username={name}")
# Check the status code of the response
if response.status_code != 200:
print(f"Failed to get data for {name}: {response.status_code}")
continue
# Try to parse the response as JSON
try:
user_data = response.json()
except ValueError:
print(f"Failed to parse JSON for {name}")
continue
# Extract the necessary information from the response
user_id = user_data["Id"]
username = user_data["Username"]
avatar_uri = user_data["AvatarUri"]
avatar_final = user_data["AvatarFinal"]
is_online = user_data["IsOnline"]
# Add the user's information to the dictionary
users[user_id] = {
"Id": user_id,
"Username": username,
"AvatarUri": avatar_uri,
"AvatarFinal": avatar_final,
"IsOnline": is_online
}
# Save the dictionary to a JSON file
with open("users.json", "w") as f:
json.dump(users, f)

You can often overcome this with judicious use of proxies.
Start by getting a list of proxies from which you will make random selections. I have a scraper that acquires proxies from https://free-proxy-list.net
My list (extract) looks like this:-
http://103.197.71.7:80 - no
http://163.116.177.33:808 - yes
'yes' means that HTTPS is supported. This list currently contains 95 proxies. It varies depending on the response from my scraper
So we start by parsing the proxy list. Subsequently we choose proxies at random before trying to access the Roblox API. This may not run quickly because the proxies are not necessarily reliable. They are free after all.
from requests import get as GET, packages as PACKAGES
from random import choice as CHOICE
from concurrent.futures import ThreadPoolExecutor as TPE
PACKAGES.urllib3.util.ssl_.DEFAULT_CIPHERS = 'ALL:#SECLEVEL=1'
ROBLOX_API = 'https://api.roblox.com/users/get-by-username'
TIMEOUT = 1
def get_proxies():
http, https = list(), list()
with open('proxylist.txt') as p:
for line in p:
proxy_url, _, supports_https = line.split()
_list = https if supports_https == 'yes' else http
_list.append(proxy_url)
return http, https
http, https = get_proxies()
def process(name):
params = {'username': name.strip()}
while True:
try:
proxy = {'http': CHOICE(http), 'https': CHOICE(https)}
(r := GET(ROBLOX_API, params=params, proxies=proxy, timeout=TIMEOUT)).raise_for_status()
if (j := r.json()).get('success', True):
print(j)
break
except Exception as e:
pass
with open('names.txt') as names:
with TPE() as executor:
executor.map(process, names)
In principle, the while loop in process() could get stuck so it might make sense to limit the number of retries.
This produces the following output:
{'Id': 4082578648, 'Username': 'paquita12345jeje', 'AvatarUri': None, 'AvatarFinal': False, 'IsOnline': False}
{'Id': 2965702542, 'Username': 'mekayla_091', 'AvatarUri': None, 'AvatarFinal': False, 'IsOnline': False}
{'Id': 4079018794, 'Username': 'app_58230', 'AvatarUri': None, 'AvatarFinal': False, 'IsOnline': False}
{'Id': 3437922948, 'Username': 'kakoytochelik123', 'AvatarUri': None, 'AvatarFinal': False, 'IsOnline': False}
{'Id': 4082346906, 'Username': 'Abobausrip', 'AvatarUri': None, 'AvatarFinal': False, 'IsOnline': False}
{'Id': 2988555289, 'Username': 'HixPlayk', 'AvatarUri': None, 'AvatarFinal': False, 'IsOnline': False}
{'Id': 3286921649, 'Username': 'Sherri0708', 'AvatarUri': None, 'AvatarFinal': False, 'IsOnline': False}
{'Id': 1441252794, 'Username': 'ghfgrgt7nfdbfj', 'AvatarUri': None, 'AvatarFinal': False, 'IsOnline': False}
{'Id': 4088896225, 'Username': 'ddddorffg', 'AvatarUri': None, 'AvatarFinal': False, 'IsOnline': False}
{'Id': 3443374919, 'Username': 'TheWolfylikedog', 'AvatarUri': None, 'AvatarFinal': False, 'IsOnline': False}
{'Id': 3980932331, 'Username': 'Ameliathebest727', 'AvatarUri': None, 'AvatarFinal': False, 'IsOnline': False}
{'Id': 3773237135, 'Username': 'rip_robson0007', 'AvatarUri': None, 'AvatarFinal': False, 'IsOnline': False}
{'Id': 4082991447, 'Username': 'hfsgfhsgfhgfhds', 'AvatarUri': None, 'AvatarFinal': False, 'IsOnline': False}

Pass query params but got error that query params is required

I write some unit tests for my django app. but test failed, because of that query params missed but I passed the query params.
This is one of my tests:
def test_delete_card(self):
url = reverse("postideas-urls:DeleteCard")
card_pk=self.test_card_second.id
data3 = {
'card_pk':card_pk
}
self.client.credentials(HTTP_AUTHORIZATION='Bearer ' + self.token)
response = self.client.delete(url,params=data3)
print(response.__dict__)
self.assertEqual(response.status_code, status.HTTP_200_OK)
and this is my response.__dict___
... '_container': [b'{"error":"card_pk query param(s) are required"}'], '_is_rendered': True, 'data': {'error': 'card_pk query param(s) are required'}, 'exception': False, 'content_type': None, 'accepted_renderer': <rest_framework.renderers.JSONRenderer object at 0x7fb757f3aad0>, 'accepted_media_type': 'application/json', 'renderer_context': {'view': <postideas.views.DeleteCardView object at 0x7fb757f37d10>, 'args': (), 'kwargs': {}, 'request': <rest_framework.request.Request: DELETE '/api/v1/postideas/Delete_Card/'>, 'response': <Response status_code=400, ...'request': {'PATH_INFO': '/api/v1/postideas/Delete_Card/', 'REQUEST_METHOD': 'DELETE', 'SERVER_PORT': '80', 'wsgi.url_scheme': 'http', 'params': {'card_pk': 5}, 'QUERY_STRING': '', 'HTTP_AUTHORIZATION': 'Bearer ...

You pass the data as params=data3 but it is supposed to be either the 2nd positional argument or the keyword argument data. Hence you can make the request as:
response = self.client.delete(url, data3)
OR
response = self.client.delete(url, data=data3)
Note: Also if you use the generic views by DRF the status code for a
successful delete request is 204, hence the assert might need to be
changed to:
self.assertEqual(response.status_code, status.HTTP_204_NO_CONTENT)

Using python to get a track from the spotify API by using search Endpoint

So I'm trying to get a track from the spotify API by searching for it by using the search endpoint of the API (See documentation). First, I authorize myself so I can send GET requests. This happens without issues, I added the code for reproducibility.
import requests
CLIENT_ID = "your_id_here"
CLIENT_SECRET = "your_secret_here"
AUTH_URL = "https://accounts.spotify.com/api/token"
auth_response = requests.post(AUTH_URL, {
'grant_type': 'client_credentials',
'client_id': CLIENT_ID,
'client_secret': CLIENT_SECRET,
})
#Convert response to JSON
auth_response_data = auth_response.json()
#Save the access token
access_token = auth_response_data['access_token']
#Need to pass access token into header to send properly formed GET request to API server
headers = {
'Authorization': 'Bearer {token}'.format(token=access_token)
}
Then, I want to use the search endpoint of the API to find a track by using the track name + artist (I need the track ID later on). When I use the example provided in the documentation, everything works fine and an artist object is returned by using the following query:
BASE_URL = 'https://api.spotify.com/v1/'
r = requests.get(BASE_URL + 'search?q=tania%20bowra&type=artist', headers=headers)
r = r.json()
This is the response, which looks exactly like the one in documentation:
{'artists': {'href': 'https://api.spotify.com/v1/search?query=tania+bowra&type=artist&offset=0&limit=20',
'items': [{'external_urls': {'spotify': 'https://open.spotify.com/artist/08td7MxkoHQkXnWAYD8d6Q'},
'followers': {'href': None, 'total': 235},
'genres': [],
'href': 'https://api.spotify.com/v1/artists/08td7MxkoHQkXnWAYD8d6Q',
'id': '08td7MxkoHQkXnWAYD8d6Q',
'images': [{'height': 640,
'url': 'https://i.scdn.co/image/ab67616d0000b2731ae2bdc1378da1b440e1f610',
'width': 640},
{'height': 300,
'url': 'https://i.scdn.co/image/ab67616d00001e021ae2bdc1378da1b440e1f610',
'width': 300},
{'height': 64,
'url': 'https://i.scdn.co/image/ab67616d000048511ae2bdc1378da1b440e1f610',
'width': 64}],
'name': 'Tania Bowra',
'popularity': 1,
'type': 'artist',
'uri': 'spotify:artist:08td7MxkoHQkXnWAYD8d6Q'}],
'limit': 20,
'next': None,
'offset': 0,
'previous': None,
'total': 1}}
Applying the same logic, I tried to get a track object from the api by using an artist and a track name, like so:
BASE_URL = 'https://api.spotify.com/v1/'
r = requests.get(BASE_URL + 'search?q=artist:queen%20track:bohemian%20rapsody&type=track', headers=headers)
r = r.json()
Even though I do get a valid response (statuscode==200), it seems to be empty:
{'tracks': {'href': 'https://api.spotify.com/v1/search?query=artist%3Aqueen+track%3Abohemian+rapsody&type=track&offset=0&limit=20',
'items': [],
'limit': 20,
'next': None,
'offset': 0,
'previous': None,
'total': 0}}
My question is: why is this happening?

You are now searching for the query: artist:queen%20track:bohemian%20rapsody while this should just be queen%20bohemian%20rapsody instead. the type afterwards shows what items you want to return. You dont have to determine the artist and track name seperately in the query. Interpret the query just like typing something into the spotify search bar.

Problem solved. It was rhapsody instead of rapsody... Sucks be a non-native english speaker sometimes =)

Find oldest to youngest record Python

I am trying to retrieve data from the first available date to present date from an API. I've tried using min and max in parameter.
def getcomplete(cid, pid, tag, type):
api_endpoint = ''
headers = {'token': get_token()['access_token'], 'Content-Type': 'application/json'}
params = {'cid': str(cid),
'from-date': datetime.datetime.min,
'to-date': datetime.datetime.max,
'tag': str(tag),
'type': str(type),
'pid': str(pid)
}
r = requests.post(url=api_endpoint, headers=headers, params=params)
return r.json()
getcomplete(10,12,'x','y')
This returns {'status': 'success', 'message': 'success', 'data': []}.
Is there anything wrong with the written function.
Thanks

Pythons min() and max() have an optional default parameter. This will prevent them from throwing errors
min("", default="")

Getting error 404 while trying to update a value in Rest API of Openhab using Python request.put

I am trying to update a value in REST API of openhab using requests.put in Python. But I am getting error 404.
My code is copied below
import requests
import json
from pprint import pprint
TemperatureA_FF_Office = 20
headers = {'Content-type': 'application/json'}
payload = {'state' : TemperatureA_FF_Office}
payld = json.dumps(payload)
re = requests.put("http://localhost:8080/rest/items/TemperatureA_FF_Office
/state/put", params= payld, headers = headers)
pprint(vars(re))
The error code I am getting is
{'_content': '',
'_content_consumed': True,
'connection': <requests.adapters.HTTPAdapter object at 7fd3b55ec9d0>,
'cookies': <<class 'requests.cookies.RequestsCookieJar'>[]>,
'elapsed': datetime.timedelta(0, 0, 4019),
'encoding': None,
'history': [],
'raw': <urllib3.response.HTTPResponse object at 0x7fd3b55ecd90>,
'reason': 'Not Found',
'request': <PreparedRequest [PUT]>,
'status_code': 404,
'url': u'http://localhost:8080/rest/items/TemperatureA_FF_Office/state/put?state=21.0'}
Is this the way to use requests.put? Please help.

Try something along these lines:
import requests
req = "http://localhost:8080/rest/items/YOUR_SENSOR_HERE/state"
val = VARIABLE_WITH_YOUR_SENSOR_DATA
try:
r = requests.put(req,data=val)
except requests.ConnectionError as e:
r = "Response Error"
print e
print r
This is a massively simplified version of what I'm using for some of my presence detection and temperature scripts.
The printing of 'r' and 'e' is useful for debug purposes and can be removed once you've got your script working properly.