Develop Reference

!curl commands in Python notebook fail with 500 Internal error

I am running the below code in Google Colab and get The server encountered an internal error or misconfiguration and was unable to complete your request. If I am running the command without passing in the variable $data like below, it runs perfectly fine. Only when I'm looping through the file and passing variables it seems to be failing
import csv
import json
reader = csv.reader(open('/content/drive/MyDrive/file5.csv'))
for row in reader:
data = {"snps": row[0], "pop": "YRI", "r2_threshold": "0.9", "maf_threshold": "0.01"}
data = json.dumps(data)
data = "'{}'".format(data)
!curl -k -H "Content-Type: application/json" -X POST -d "$data" 'https://ldlink.nci.nih.gov/LDlinkRest/snpclip?token=e3e559472899'
This works:
!curl -k -H "Content-Type: application/json" -X POST -d '{"snps": "rs3\nrs4", "pop":"YRI", "r2_threshold": "0.1", "maf_threshold": "0.01"}' 'https://ldlink.nci.nih.gov/LDlinkRest/snpclip?token=e3e559472899'

UPDATE: Actually, ipython does allow you to run ! escapes in a loop; the actual error in your code is purely in the incorrect quoting (especially the addition of single quotes around the data value, but there could be more).
Original (partially incorrect) answer below.
The ! escape tells your notebook (Google Colab, Jupyter, or what have you; basically whatever is running ipython as a kernel or similar) to leave Python and run a shell command. Python itself has no support for this; the closest approximation would be something like
import subprocess
...
for row in reader:
data = {"snps": row[0], "pop": "YRI", "r2_threshold": "0.9", "maf_threshold": "0.01"}
data = json.dumps(data)
# This was wrong on so many levels
# data = "'{}'".format(data)
subprocess.run(['curl', '-k',
'-H', "Content-Type: application/json",
'-X', 'POST', '-d', data,
'https://ldlink.nci.nih.gov/LDlinkRest/snpclip?token=e3e559472899'],
text=True, check=True)
though avoiding subprocess and running Python urllib or requests code to perform the POST would be more efficient and elegant, and give you more control over what gets sent and how it gets handled.
How to properly quote strings when translating between shell commands and Python requires you to understand the shell's quoting behavior. I'll just briefly note that I left double quotes where they were not incorrect in your original command, but otherwise preferred single quotes, and of course, data now refers to a proper Python variable with that name, not a shell variable with the same name.
To reiterate: ipython (which is what your notebook is an interface to) knows how to run both Python code and shell scipt code via !; but once you ask it to run Python code, ipython hands it over to Python proper, and you are no longer in ipython.

How to get data from web in python using curl?

In bash when I used
myscript.sh
file="/tmp/vipin/kk.txt"
curl -L "myabcurlx=10&id-11.com" > $file
cat $file
./myscript.sh gives me below output
1,2,33abc
2,54fdd,fddg3
3,fffff,gfr54
When I tried to fetch it using python and tried below code -
mypython.py
command = curl + ' -L ' + 'myabcurlx=10&id-11.com'
output = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE).stdout.read().decode('ascii')
print(output)
python mypython.py throw error, Can you please point out what is wrong with my code.
Error :
/bin/sh: line 1: &id=11: command not found
Wrong Parameter

command = curl + ' -L ' + 'myabcurlx=10&id-11.com'
Print out what this string is, or just think about it. Assuming that curl is the string 'curl' or '/usr/bin/curl' or something, you get:
curl -L myabcurlx=10&id-11.com
That’s obviously not the same thing you typed at the shell. Most importantly, that last argument is not quoted, and it has a & in the middle of it, which means that what you’re actually asking it to do is to run curl in the background and then run some other program that doesn’t exist, as if you’d done this:
curl -L myabcurlx=10 &
id-11.com
Obviously you could manually include quotes in the string:
command = curl + ' -L ' + '"myabcurlx=10&id-11.com"'
… but that won’t work if the string is, say, a variable rather than a literal in your source—especially if that variable might have quote characters within it.
The shlex module has helpers to quoting things properly.
But the easiest thing to do is just not try to build a command line in the first place. You aren’t using any shell features here, so why add the extra headaches, performance costs, problems with the shell getting in the way of your output and retcode, and possible security issues for no benefit?
Make the arguments a list rather than a string:
command = [curl, '-L', 'myabcurlx=10&id-11.com']
… and leave off the shell=True
And it just works. No need to get spaces and quotes and escapes right.
Well, it still won’t work, because Popen doesn’t return output, it’s a constructor for a Popen object. But that’s a whole separate problem—which should be easy to solve if you read the docs.
But for this case, an even better solution is to use the Python bindings to libcurl instead of calling the command-line tool. Or, even better, since you’re not using any of the complicated features of curl in the first place, just use requests to make the same request. Either way, you get a response object as a Python object with useful attributes like text and headers and request.headers that you can’t get from a command line tool except by parsing its output as a giant string.

import subprocess
fileName="/tmp/vipin/kk.txt"
with open(fileName,"w") as f:
subprocess.read(["curl","-L","myabcurlx=10&id-11.com"],stdout=f)
print(fileName)
recommended approaches:
https://docs.python.org/3.7/library/urllib.request.html#examples
http://docs.python-requests.org/en/master/user/install/

Newlines removed in POST request body? (Google App Engine)

I am building a REST API on Google App Engine (not using Endpoints) that will allow users to upload a CSV or tab-delimited file and search for potential duplicates. Since it's an API, I cannot use <form>s or the BlobStore's upload_url. I also cannot rely on having a single web client that will call this API. Instead, ideally, users will send the file in the body of the request.
My problem is, when I try to read the content of a tab-delimited file, I find that all newline characters have been removed, so there is no way of splitting the content into rows.
If I check the content of the file directly on the Python interpreter, I see that tabs and newlines are there (output is truncated in the example)
>>> with open('./data/occ_sample.txt') as o:
... o.read()
...
'id\ttype\tmodified\tlanguage\trights\n123456\tPhysicalObject\t2015-11-11 11:50:59.0\ten\thttp://creativecommons.org/licenses/by-nc/3.0\n...'
The RequestHandler logs the content of the request body:
import logging
class ReportApi(webapp2.RequestHandler):
def post(self):
logging.info(self.request.body)
...
So when I call the API running in the dev_appserver via curl
curl -X POST -d #data/occ_sample.txt http://localhost:8080/api/v0/report
This shows up in the logs:
id type modified language rights123456 PhysicalObject 2015-11-11 11:50:59.0 en http://creativecommons.org/licenses/by-nc/3.0
As you can see, there is nothing between the last value of the headers and the first record (rights and 123456 respectively) and the same happens with the last value of each record and the first one of the next.
Am I missing something obvious here? I have tried loading the data with self.request.body, self.request.body_file and self.request.POST, and none seem to work. I also tried applying the Content-Type values text/csv, text/plain, application/csv in the request headers, with no success. Should I add a different Content-Type?

You are using the wrong curl command-line option to send your file data, and it is this option that is stripping the newlines.
The -d option parses out your data and sends a application/x-www-form-urlencoded request, and it strips newlines. From the curl manpage:
-d, --data <data>
[...]
If you start the data with the letter #, the rest should be a file name to read the data from, or - if you want curl to read the data from stdin. Multiple files can also be specified. Posting data from a file named 'foobar' would thus be done with --data #foobar. When --data is told to read from a file like that, carriage returns and newlines will be stripped out.
Bold emphasis mine.
Use the --data-binary option instead:
--data-binary <data>
(HTTP) This posts data exactly as specified with no extra processing whatsoever.
If you start the data with the letter #, the rest should be a filename. Data is posted in a similar manner as --data-ascii does, except that newlines and carriage returns are preserved and conversions are never done.
You may want to include a Content-Type header in that case; of course this depends on your handler if you care about that header.

NOAA Weather REST API causes error when requesting with curl

I am trying to write a python program using NOAA's Climate Data Online REST Web Services (http://www.ncdc.noaa.gov/cdo-web/webservices/v2#data). But, I am running into errors in my request responses. When attempting a request with curl from command line I input:
curl -H "token:<MYTOKEN>" http://www.ncdc.noaa.gov/cdo-web/api/v2/data?datasetid=GHCND&locationid=ZIP:22405&startdate=1999-10-05&enddate=1999-10-25
It returns this response:
[1] 24322
[2] 24323
[3] 24324
phil#philUbu:~$ <?xml version="1.0" encoding="UTF-8" standalone="yes"?><response><statusCode>400</statusCode><userMessage>There was an error with the request.</userMessage><developerMessage>Required parameter 'startdate' is missing.</developerMessage></response>
[1] Done curl -H "token:..." http://www.ncdc.noaa.gov/cdo-web/api/v2/data?datasetid=GHCND
[2]- Done locationid=ZIP:22405
[3]+ Done startdate=1999-10-05
For some reason it thinks I am missing the startdate, but I have included it and it is in the proper format according to the documentation. Does anybody have any ideas of what the problem could be?

The ampersands in the url are probably being parsed by your shell. Put single quotes around it:
curl -H "token:<MYTOKEN>" 'http://www.ncdc.noaa.gov/cdo-web/api/v2/data?datasetid=GHCND&locationid=ZIP:22405&startdate=1999-10-05&enddate=1999-10-25'

python subprocess.popen redirect to create a file

I've been searching for how to do this without any success. I've inherited a python script for performing an hourly backup on our database. The original script is too slow, so I'm trying a faster utility. My new command would look like this if typed into a shell:
pg_basebackup -h 127.0.0.1 -F t -X f -c fast -z -D - --username=postgres > db.backup.tgz
The problem is that the original script uses call(cmd) and it fails if the above is the cmd string. I've been looking for how to modify this to use popen but cannot find any examples where a file create redirect is used as in
>. The pg_basebackup as shown will output to stdout. The only way I've succeeded so far is to change -D - to -D some.file.tgz and then move the file to the archive, but I'd rather do this in one step.
Any ideas?
Jay

May be like this ?
with open("db.backup.tgz","a") as stdout:
p = subprocess.Popen(cmd_without_redirector, stdout=stdout, stderr=stdout, shell=True)
p.wait()

Hmmm... The pg_basebackup executable must be able to attach to that file. If I open the file in the manner you suggest, I don't know the correct syntax in python to be able to do that. If I try putting either " > " or " >> " in the string to call with cmd(), python pukes on it. That's my real problem that I'm not finding any guidance on.