Develop Reference

Python script - API call fails due to JSON arguments

I am trying to pull reports through a (shadowserver) API - doc here:
The python script is written by ShadowServer and I have tested it with the simple usage example they are providing:
$ ./call-api.py test/ping '{}'
{"pong":"2020-10-26 23:06:37"}
Thus far, everything works fine. I can submit additional calls and I get a response back.
However, when i try to pass a JSON parameter to the script (that is executed in command prompt), such as the example below:
os.system('''[some_directory]\\call-api.py reports/query '{ "help":true }' pretty''')
I get the following result:
JSON Exception: Expecting value: line 1 column 1 (char 0)
I believe that it is an issue with the quotes - single/double/triple.. but I've tried everything and it keeps returning the same result.
Any ideas?
Many thanks!

You might want to try subprocess to see if you get the same result, e.g. like this
import subprocess
subprocess.Popen([
"[some_directory]\\call-api.py",
"reports/query",
'{"help": true}',
"pretty",
])
or use json.dumps to not worry about any quoting
import subprocess
import json
subprocess.Popen([
"[some_directory]\\call-api.py",
"reports/query",
json.dumps({"help": True}),
"pretty",
])
SitiSchu came up with the json.dumps idea.

!curl commands in Python notebook fail with 500 Internal error

I am running the below code in Google Colab and get The server encountered an internal error or misconfiguration and was unable to complete your request. If I am running the command without passing in the variable $data like below, it runs perfectly fine. Only when I'm looping through the file and passing variables it seems to be failing
import csv
import json
reader = csv.reader(open('/content/drive/MyDrive/file5.csv'))
for row in reader:
data = {"snps": row[0], "pop": "YRI", "r2_threshold": "0.9", "maf_threshold": "0.01"}
data = json.dumps(data)
data = "'{}'".format(data)
!curl -k -H "Content-Type: application/json" -X POST -d "$data" 'https://ldlink.nci.nih.gov/LDlinkRest/snpclip?token=e3e559472899'
This works:
!curl -k -H "Content-Type: application/json" -X POST -d '{"snps": "rs3\nrs4", "pop":"YRI", "r2_threshold": "0.1", "maf_threshold": "0.01"}' 'https://ldlink.nci.nih.gov/LDlinkRest/snpclip?token=e3e559472899'

UPDATE: Actually, ipython does allow you to run ! escapes in a loop; the actual error in your code is purely in the incorrect quoting (especially the addition of single quotes around the data value, but there could be more).
Original (partially incorrect) answer below.
The ! escape tells your notebook (Google Colab, Jupyter, or what have you; basically whatever is running ipython as a kernel or similar) to leave Python and run a shell command. Python itself has no support for this; the closest approximation would be something like
import subprocess
...
for row in reader:
data = {"snps": row[0], "pop": "YRI", "r2_threshold": "0.9", "maf_threshold": "0.01"}
data = json.dumps(data)
# This was wrong on so many levels
# data = "'{}'".format(data)
subprocess.run(['curl', '-k',
'-H', "Content-Type: application/json",
'-X', 'POST', '-d', data,
'https://ldlink.nci.nih.gov/LDlinkRest/snpclip?token=e3e559472899'],
text=True, check=True)
though avoiding subprocess and running Python urllib or requests code to perform the POST would be more efficient and elegant, and give you more control over what gets sent and how it gets handled.
How to properly quote strings when translating between shell commands and Python requires you to understand the shell's quoting behavior. I'll just briefly note that I left double quotes where they were not incorrect in your original command, but otherwise preferred single quotes, and of course, data now refers to a proper Python variable with that name, not a shell variable with the same name.
To reiterate: ipython (which is what your notebook is an interface to) knows how to run both Python code and shell scipt code via !; but once you ask it to run Python code, ipython hands it over to Python proper, and you are no longer in ipython.

Same command giving different results when run in CLI and when as an argument for os.system()

I'm using python to try and scrape twitter data using SNSCRAPE, and for that I need to run the following example command to get the output I want:
snscrape --max-results 100 twitter-search '#Tesla #Elonmusk' > scrapedtweets.txt
When run on the terminal, it sends the output directly to a text file named 'scrapedtweets'. But when I plug this string into os.system like so:
os.system("snscrape --max-results 100 twitter-search '#Tesla #Elonmusk' > scrapedtweets.txt")
I get an error code '2', and scrapedtweets.txt remains blank.
I should mention that the command works perfectly in os.system() when there are no # symbols involved in the command, which leads me to think that it is being parsed funny by the function, but I have no idea what the exact problem is or how to solve it.
Or alternatively is there something wrong with how I am using os.system()?
Any and all help is appreciated.

NOAA Weather REST API causes error when requesting with curl

I am trying to write a python program using NOAA's Climate Data Online REST Web Services (http://www.ncdc.noaa.gov/cdo-web/webservices/v2#data). But, I am running into errors in my request responses. When attempting a request with curl from command line I input:
curl -H "token:<MYTOKEN>" http://www.ncdc.noaa.gov/cdo-web/api/v2/data?datasetid=GHCND&locationid=ZIP:22405&startdate=1999-10-05&enddate=1999-10-25
It returns this response:
[1] 24322
[2] 24323
[3] 24324
phil#philUbu:~$ <?xml version="1.0" encoding="UTF-8" standalone="yes"?><response><statusCode>400</statusCode><userMessage>There was an error with the request.</userMessage><developerMessage>Required parameter 'startdate' is missing.</developerMessage></response>
[1] Done curl -H "token:..." http://www.ncdc.noaa.gov/cdo-web/api/v2/data?datasetid=GHCND
[2]- Done locationid=ZIP:22405
[3]+ Done startdate=1999-10-05
For some reason it thinks I am missing the startdate, but I have included it and it is in the proper format according to the documentation. Does anybody have any ideas of what the problem could be?

The ampersands in the url are probably being parsed by your shell. Put single quotes around it:
curl -H "token:<MYTOKEN>" 'http://www.ncdc.noaa.gov/cdo-web/api/v2/data?datasetid=GHCND&locationid=ZIP:22405&startdate=1999-10-05&enddate=1999-10-25'

Pycurl redirect option ignored and assertion failed trying to read video from the web?

I am trying to write a program that reads a webpage looking for file links, which it then attempts to download using curl/libcurl/pycurl. I have everything up to the pycurl correctly working, and when I use a curl command in the terminal, I can get the file to download. The curl command looks like the following:
curl -LO https://archive.org/download/TheThreeStooges/TheThreeStooges-001-WomanHaters1934moeLarryCurleydivxdabaron19m20s.mp4
This results in one redirect (a file that reads as all 0s on the output) and then it correctly downloads the file. When I remove the -L flag (so the command is just -O) it only reaches the first line, where it doesn't find a file, and stops.
But when I try to do the same operation using pycurl in a Python script, I am unable to successfully set [Curl object].FOLLOWLOCATION to 1, which is supposed to be the equivalent of the -L flag. The python code looks like the following:
c = [class Curl object] # get a Curl object
fp = open(file_name,'wb')
c.setopt(c.URL , full_url) # set the url
c.setopt(c.FOLLOWLOCATION, 1)
c.setopt(c.WRITEDATA , fp)
c.perform()
When this runs, it gets to c.perform() and shows the following:
python2.7: src/pycurl.c:272: get_thread_state: Assertion `self->ob_type == p_Curl_Type' failed.
Is it missing the redirect, or am I missing something else earlier because I am relatively new to cURL?

When I enabled verbose output for the c.perform() step, I was able to uncover what I believe was/is the underlying problem that my program had. The first line, which was effectively flagged, indicated that an open connection was being reused.
I had originally packaged the file into an object oriented setup, as opposed to a script, so the curl object had been read and reused without being closed. Therefore after the first connection attempt, which failed because I didn't set options correctly, it was reusing the connection to the website/server (which presumably had the wrong connection settings).
The problem was resolved by having the script close any existing Curl objects, and create a new one before the file download.