Develop Reference

Download xlsx file with Python

Want to download to local directory. This code works for csv but not xlsx. It writes a file but cannot be opened as Excel.
Any help will be appreciated.
url = 'https://some_url'
resp = requests.get(url)
open('some_filename.xlsx', 'wb').write(resp.content)

You could create a dataframe from the resp data and then use pd.to_excel() function to obtain the xlsx file. This is a tested solution, and it worked for me.
import requests
import pandas as pd
import io
url='https://www.google.com' #as an example
urlData = requests.get(url).content #Get the content from the url
dataframe = pd.read_csv(io.StringIO(urlData.decode('latin-1')))
filename="data.xlsx"
dataframe.to_excel(filename)

In pandas you could just do:
import pandas as pd
url = 'https://some_url'
df = pd.read_csv(url)

Python-How to send a dataframe to external API (push)

I am wondering how to send this dataframe to external API?

You can convert your dataframe into json file using pandas.to_json(), read that file and pass it as data in the post call of requests.
df_json_dict = json.loads(df.to_json(orient='records'))
requests.post(url, data=df_json_dict)

In case the answer is still not clear, I will summarise the thing:
create your pandas dataframe
import pandas as pd
dataframe = pd.read_csv(file_path, sep=',')
create the stream
import io, requests
stream = io.StringIO()
convert dataframe to csv stream
dataframe.to_csv(stream, sep=';', encoding='utf-8', index = False)
start stream from the beginning
stream.seek(0)
put the request using stream
file_upload_resp = requests.put(url, data=stream)

How to convert a large json response of a web service call into CSV in Python?

I have a web service which returns a very large JSON response. I want to parse it and convert it into a CSV format using Python. I have written a code to load json and convert it to CSV. However, for a large response it raises MemoryError. How can I load and convert response data using streaming?
Here is my code:
import json
from pandas import json_normalize
re = requests.get(url)
data = json.loads(re.text)
df = json_normalize(data)
df.to_csv(fileName, index=False, encoding='utf-8')
Here is a sample of my JSON response:
[{"F1":"V1_1","F2":false,,"F3":120,"F4":"URL1","F5":{"F5_1":4,"F5_2":"A"}},
{"F1":"V2_1","F2":true,,"F3":450,"F4":"URL2","F5":{"F5_1":13,"F5_2":"B"}},
{"F1":"V3_1","F2":false,,"F3":312,"F4":"URL3","F5":{"F5_1":6,"F5_2":"C"}},
...
]
The MemoryError occurs in the json.loads() function (or the response.json() function if use it instead of json.loads()). Is there any idea how I can load and parse and convert such a big JSON response to a CSV file?

First, you are not making dataframe from only the result of the response, rather you are trying to push additional data with the result here. Try this and see if pandas can save this time.
import pandas as pd
# this is a dummy URL for demonstration
url = "https://www.qnt.io/api/results?pID=gifgif&mID=54a309ae1c61be23aba0da62&key=54a309ac1c61be23aba0da3f"
response = requests.get(url)
# extract the relevant results from response
data = response.json()["results"]
df = pd.json_normalize(data)
df.to_csv("filename.csv", index=False, encoding="utf-8")

Python: read a csv file generated dynamically by an API?

I want to read into pandas the csv generated by this URL:
https://www.alphavantage.co/query?function=FX_DAILY&from_symbol=EUR&to_symbol=USD&apikey=demo&datatype=csv
How should this be done?

I believe you can just read it with pd.read_csv
import pandas as pd
URL = 'https://www.alphavantage.co/query?function=FX_DAILY&from_symbol=EUR&to_symbol=USD&apikey=demo&datatype=csv'
df = pd.read_csv(URL)
Results:

Convert text data from requests object to dataframe with pandas

Using requests I am creating an object which is in .csv format. How can I then write that object to a DataFrame with pandas?
To get the requests object in text format:
import requests
import pandas as pd
url = r'http://test.url'
r = requests.get(url)
r.text #this will return the data as text in csv format
I tried (doesn't work):
pd.read_csv(r.text)
pd.DataFrame.from_csv(r.text)

Try this
import requests
import pandas as pd
import io
urlData = requests.get(url).content
rawData = pd.read_csv(io.StringIO(urlData.decode('utf-8')))

I think you can use read_csv with url:
pd.read_csv(url)
filepath_or_buffer : str, pathlib.Path, py._path.local.LocalPath or any object with a read() method (such as a file handle or StringIO)
The string could be a URL. Valid URL schemes include http, ftp, s3, and file. For file URLs, a host is expected. For instance, a local file could be file ://localhost/path/to/table.csv
import pandas as pd
import io
import requests
url = r'http://...'
r = requests.get(url)
df = pd.read_csv(io.StringIO(r))
If it doesnt work, try update last line:
import pandas as pd
import io
import requests
url = r'http://...'
r = requests.get(url)
df = pd.read_csv(io.StringIO(r.text))

Using "read_csv with url" worked:
import requests, csv
import pandas as pd
url = 'https://arte.folha.uol.com.br/ciencia/2020/coronavirus/csv/mundo/dados-bra.csv'
corona_bra = pd.read_csv(url)
print(corona_bra.head())

if the url has no authentication then you can directly use read_csv(url)
if you have authentication you can use request to get it un-pickel and print the csv and make sure the result is CSV and use panda.
You can directly use importing
import csv