I have a Python script that I want to upload to our server so that it can run every day, and part of the script's function is to download a file from an SFTP connection, do some work on it and upload it to our Amazon S3 bucket.* However, I want my program to be as lean as possible so that it doesn't mess with the other daily tasks we have on the server, so I want to use only REST services and not import anything. I want to get my requests working in a REST client (I've been using Insomnia) before I put them into code.
I've searched high and low to find tutorials for how to do this, but I've found nothing. Amazon's S3 docs say things like: "Request syntax: GET / HTTP/1.1", but that doesn't tell me anything about how to actually make the request or even what URL to use to make the request to.
Can anyone give me some guidance (or at least a URL)? Thanks!
*Edit: As pointed out by Martin Prikryl in the comments, I cannot make SFTP requests using REST. I still want to find out how to make S3 requests though.
Related
Im creating a server in Python that receives POST requests, process the information in the request using some scripts (sometimes using a database) and send back a answer in JSON format. Im searching for a way to run this server and code in the cloud, in a way that i dont need my PC turned on for it to work, because my connection is very unstable.
There are a lot of web hosting companies out there, you just need to find the one that is right for you.
My personal favorite for python apps is heroku, but there are many out there. AWS is another popular one.
In future when asking questions, try to do more research before hand, and try to be more specific with questions. It would have been useful to know what kind of database you are using, or whether you're using flask or django.
Example URL = https://westgate-production-4cb87.firebaseapp.com/super-contests/weekly-card/embed
I've written the code needed to get this to work with the current deployed websocket URL and successfully scrape the data on this page -
wss://s-usc1c-nss-276.firebaseio.com/.ws?v=5&ns=westgate-production-4cb87
However, it seems that every so often the number in the subdomain will change (in this example - 276). I can obviously manually figure out what the new URL is using the Network tab in Dev Tools, but I was wondering if there was a Python code snippet or some approach where I can programatically detect what websockets are opening so that I can capture the wss:// URL and then pass it into the rest of my code that works correctly.
Thank you!
You're using an undocumented way to access the Firebase Realtime Database.
That first part of the URL is the server that your client is connecting to to access the data, and is actually determined by the first request(s) when the connection is established. But as said, this knowledge is not documented and can thus change at any time without notice.
I'd recommend using either the Python SDK, the REST API, or the REST Streaming API to access the database, as all of those are properly documented.
My question may look silly but I am asking this after too much search on Google, yet not have any clue.
I am using iCloud web services. For that I have converted this Python code to PHP. https://github.com/picklepete/pyicloud
Up to this, everything is working good. When authenticate using icloud username,password I am getting a list of web service URLs as part of response. Now for example to use Contacts web service, I need to use Contact web service URL and add a part to that URL to fetch contacts.
https://p45-contactsws.icloud.com:443/co/startup with some parameters.
The webservice URL https://p45-contactsws.icloud.com:443 is coming in response while authenticating. But the later part, 'co/startup' is there in the python code. I don't know how they found that part. So for some services which is there in Python code, they are working good. But I want to use few other service like https://p45-settingsws.icloud.com:443, https://p45-keyvalueservice.icloud.com:443 etc. and when I try to send request with correct parameters to this other services, I am getting errors like 404 not found or unauthorized access. So I believe that some URL part must be added to this just like contacts. If someone knows how or where can I get correct URL part, I will be really thankful.
Thanks to all in advance for their time reading/answering my question.
I am afraid there doesn't seem to be an official source for these API endpoints, since they seem to be discovered through sniffing the network calls rather than a proper guide from Apple. For example, this presentation, which comes from a forensic tools company, is from 2013 and covers some of the relevant endpoints. Note that iOS was still at versions 5 & 6 then (vs. the current v9.3).
All other code samples on the net basically are using the same set of API endpoints that were originally observed in 2012-2013. (Here's a snippet from another python module with additional URLs you may use.) However, all of them pretty much point to each other as the source.
If you'd like to pursue a different path, Apple now promotes the CloudKit and CloudKit JS solutions for registered apps working with iCloud data.
I need to make an outbound ftp request to retrieve a number of small
files. There are 6 files each less than 10K and I only need to
retrieve them once every couple of hours.
When I try to do this with urllib2.urlopen("ftp://xxx.xxx.xxx") I get
an exception AttributeError: 'module' object has no attribute
'FTP_PORT'.
I have read through the documentation and see you are only allowed to
make http and https requests from the appengine, unfortunately my
application needs to consume the ftp data, does this requirement mean
I can't use the appengine at all ? I sincerely hope not.
So has anyone else here found a way to make ftp requests, perhaps with
a paid account ? And if not what have other people chosen to do ?
does azure or ec2 allow outbound ftp requests ?
You're correct. Google App Engine does not allow you to make FTP requests. Not even with a paid account.
I had to use a LAMP instance on EC2 that handles FTP'ing through CURL, and make http requests to it from GAE.
This limitation used to drive me nuts; implementing the overhead around dynamically instantiating EC2 slave workers to relay FTP data felt like a real waste of time. Fortunately, as of April 9 this year (SDK 1.7.7) this isn't a problem any longer. Outbound sockets (e.g. FTP) are generally available to all billing-enabled apps.
Sockets API Overview (Python): https://developers.google.com/appengine/docs/python/sockets/
drivehq.com is another option. It provides both a web+ftp server. So a third party I needed to interface with (that spoke only FTP) would upload files via FTP. And then I would urlfetch them on appengine.
My (Python) AppEngine program fetches a web page from another site to scrape data from it -- but it seems like the 3rd party site is blocking requests from Google App Engine! -- I can fetch the page from development mode, but not when deployed.
Can I get around this by using a free proxy of some sort?
Can I use a free proxy to hide the fact that I am requesting from App Engine?
How do I find/choose a proxy? -- what do I need? -- how do I perform the fetch?
Is there anything else I need to know or watch out for?
Probably the correct approach is to request permission from the owners of the site you are scraping.
Even if you use a proxy, there is still a big chance that requests coming through the proxy will end up blocked as well.
Have you considered changing the user-agent?
result = urlfetch.fetch(u,headers = {'User-Agent': "Mozilla/5.0"},allow_truncated=True)
The API will always append "AppEngine-Google;" to the user-agent, but this might work if the restriction is not based on a IP address range.
What you are talking about is a valid bug in app engine sdk. Have a look at http://code.google.com/p/googleappengine/issues/detail?id=544 for bug updates, and workarounds for java and python.
I'm currently having the same problem and i was thinking about this solution (not yet tried) :
-> develop an app that fetch what you want
-> run it locally
-> fetch your local server from your initial
so the proxy is your computer which you know as not blocked
Let me know if it's works !
Well to be fair, if they don't want you doing that then you probably shouldn't. It's not nice to be mean.
But if you really want to do it, the best approach would be creating a simple proxy script and running it on a VPS or some computer with a decent enough connection.
Basically you expose a REST API from your server to your GAE, then the server just makes all the same requests it gets to the target site and returns the output.