I'm having the exact same issue as another unanswered post, but I'm willing to give whatever code/setup needed to get the question answered properly.
Like in the post I mentioned above, I am also trying to deploy files to S3 with the AWS CLI and I receive the same error:
upload failed: ... An HTTP Client raised and unhandled exception: unknown encoding: idna
I have the newest version of Python and the AWS CLI. I can get the Python shell to import encodings.idna, but the AWS CLI process boots its own shell to run the commands I assume. Which may mean that I need to somehow inject the import statement into the AWS CLI process. I've tried to edit the aws.cmd programs (one in /bin and one in /scripts), but nearly every change stopped the program from working properly.
I'm not sure what to post that can help determine what my issue is, so please let me know.
I just saw that I left this question unanswered. So, here's what I did to fix the issue.
Shortly after posting this question, I tried the AWS CLI Windows Installer once more and it worked. I still don't know why the initial installation didn't work, but I am able to upload to S3 via CLI without Python encoding errors.
Related
Silly newb question, it seems I can't use git to install content I need in the Heroku console, but my app/bot is dependent on content I normally use that for. I know very little about how git and pip work, or the right terminology to ask a question like this, so bare with me!
I have a bot made with Tomer8007's Kik Bot API that I've embedded in Flask and want to deploy to Heroku. I've deployed Flask sites there before, they work like a charm, but because I import everything to Heroku via git using the Heroku CLI I can't import this one which is already using git.
I normally use these two commands to fetch and install the dependencies I need for that project:
git clone -b new https://github.com/tomer8007/kik-bot-api-unofficial
pip install ./kik-bot-api-unofficial
I tried manually downloading and installing the API without git, but then it throws this error when I tried to run it: "TypeError: Couldn't build proto file into descriptor pool: duplicate file name (google/protobuf/descriptor.proto)", wheras it works perfectly fine when I use the two above commands instead. (This is locally btw).
I also made an attempt to import to heroku before using those commands and instead using them in the heroku console, but it throws a bunch of errors when I try. I also can't import it after using those commands locally, because I already used git. (I'm not sure how that works though, that's why I'm here.)
Everything in procfile.txt, requirements.txt, runtime.txt, etc is fine, the only issue is getting the API for the Kik bot. I've no idea what's going haywire when I attempt to manually download it instead of using git clone, or what alternative options I have. Any pointers?
Apparently, the duplicate file name (google/protobuf/descriptor.proto) error was caused by a breaking change in the protobuf package in versions greater than 4.
This is now fixed, and should also work on Python 3.9+.
I am trying to set up ezsheets for the use with Google Sheets. I followed the instructions from here https://ezsheets.readthedocs.io/en/latest/ and here https://automatetheboringstuff.com/2e/chapter14/
The set up process works quite differently on my computer: Somehow I could download the credentials-sheets.json. I need to download the token-sheets.pickle and token-drive.pickle files. When I run import ezsheets, no browser window is opended as described in the set up instructions. Nothing happens.
Is there another way to download both files?
I followed the steps you referenced and managed to generate the files, but I also encountered the same issue before figuring out the cause. The problem is that there are a few possible causes and the script silently fails without telling you exactly what happened.
Here are a few suggestions:
First off you need to configure your OAuth Consent Screen. You won't be able to create the credentials without it.
Make sure that you have the right credentials file. To generate it you have to go to the Credentials page in the Cloud Console. The docs say that you need an OAuth Client ID. Make sure that you have chosen the correct app at the top.
Then you will be prompted to choose an application type. According to the docs you shared the type should be "Other", but this is no longer available so "Desktop app" is the best equivalent if you're just running a local script.
After that you can just choose a name and create the credentials. You will be prompted to download the file afterwards.
Check that the credentials-sheets.json file has that exact name.
Make sure that the credentials-sheets.json file is located in the same directory where you're running your python script file or console commands.
Check that you've enabled both the Sheets and Drive API in your GCP Project.
Python will try to setup a temporary server on http://localhost:8080/ to retrieve the pickle files. If another application is using port 8080 then it will also fail. In my case a previously failed Python script was hanging on to that port.
To find and close the processes using port 8080 you can refer to this answer for Linux/Mac or this other answer for Windows. Just make sure that the process is not something you're currently using.
I just used the single import ezsheets command to get the files so after getting the token-sheets.pickle I had to run it again to get the token-drive.pickle, but after that the library should detect that you already have the files.
I'm trying to set up an MWAA Airflow 2.0 environment that integrates S3 and GCP's Pub/Sub. While we have no problems with the environment being initialized, we're having trouble installing some dependencies and importing Python packages -- specifically apache-airflow-providers-google==2.2.0.
We've followed all of the instructions based on the official MWAA Python documentation. We already included the constraints file as prescribed by AWS, activated all Airflow logging configs, and tested the requirements.txt file using the MWAA local runner. The result when updating our MWAA environment's requirements would always be like this
When testing using the MWAA local runner, we observed that using the requirements.txt file with the constraints still takes forever to resolve. Installation takes more than 10-30 minutes which is no good.
As an experiment, we tried using a version of the requirements.txt file that omits the constraints and pinned versioning. Doing so installs the packages successfully and we don't receive import errors anymore on both MWAA local runner and our MWAA environment itself. However, all of our dags will fail to run no matter what. Airflow logs are also inaccessible whenever we do this.
The team and I have been trying to get MWAA environments up and running for our different applications and ETL pipelines but we just can't seem to get things to work smoothly. Any help would be appreciated!
I'm having the same problems and in the end we had to refactor a lot of things to remove the dependence. It looks like is a problem with PIP resolver and apache-airflow-providers-google if you look the official page:
https://pypi.org/project/apache-airflow-providers-google/2.0.0rc1/
In the WORST case, you may need to use Airflow direct on EC2 from docker image and abandon MWAA :(
I've been through similar issues but with different packages. There are certain things you need to take into consideration when using MWAA. I didn't have any issue testing the packages on the local runner then on MWAA using a public VPC, I only had issues when using a private VPC as the web server doesn't have an internet connection, so the method to get the packages to MWAA is different.
Things to take into consideration:
The version of the packages; test on the local runner if you can first
Enable the logs; The scheduler and web server logs can show you issues, but also they may not. The reason for this is Fargate serving the images, will try to roll back to a working state rather than have MWAA be in a non-working state. So, you might not see what the error actually is, it may even look like there were no errors in certain scenarios.
Check dependencies; You may need to download a package with pip download <package>==version. There you can inspect the contents of the .whl file and see if there are any dependencies. You may have extra notes that can point you in the right direction. In one case, using the Slack package wouldn't work until I also added the http package, even though Airflow includes this package.
So, yes it's serverless, and you may have an easy time installing/setting MWAA up, but be prepared to do a little investigation if it doesn't work. I did contact AWS support, but managed to solve it myself in the end. Other than trying the obvious things, only those that use MWAA frequently and have faced varying scenarios will be of any assistance.
Yesterday this code was working fine both in local and production servers:
import cloudstorage
def filelist(Handler):
gs_bucket_name="/bucketname"
list=cloudstorage.listbucket(gs_bucket_name)
logging.warning(list)
self.write(list)
for e in list:
self.write(e)
self.write("<br>")
From yesterday to today I've upgraded GAE Launcher and changed the billing options (I was using a free trial and now a paid account) (not sure if it has anything to do, but just to give extra information)
But today the code stopped working in local (works fine in production)
This is the beginning of the error log
WARNING 2015-02-20 09:50:21,721 admin.py:106] <cloudstorage.cloudstorage_api._Bucket object at 0x10ac31e90>
ERROR 2015-02-20 09:50:21,729 api_server.py:221] Exception while handling service_name: "app_identity_service"
method: "GetAccessToken"
request: "\n7https://www.googleapis.com/auth/devstorage.full_control"
request_id: "WoMrXkOyfe"
The warning shows a bucket object, but as soon as I try to iterate in the list I get the exception on the identity service.
What is hapening? Seems that I need to authorize local devserver gcs mockup, but I'm not sure how.
Remember this is only happening in devserver, not in production.
Thanks for your help
This is a problem with the latest release (1.9.18). For now, until it gets fixed, you can downgrade to 1.9.17 by downloading the installer from here and just running it: https://storage.googleapis.com/appengine-sdks/featured/GoogleAppEngineLauncher-1.9.17.dmg
As per the answer below, the 1.9.18 has been patched with a fix for this. If you still want to install the 1.9.17 version, please follow this link: https://storage.googleapis.com/appengine-sdks/deprecated/1917/GoogleAppEngineLauncher-1.9.17.dmg
It's a known bug in the dev_appserver, where the SDK does not cope with the credentials already persisted in earlier versions. For me (on Ubuntu 15.10 with SDK 1.9.33) it helped to simply remove a file:
rm ~/.config/gcloud/application_default_credentials.json
as suggested in the bug issue filed by Jari Wiklund.
Update: As of March 5th, 2105 this was fixed in the public release of 1.9.18, which may be a simpler way to get the fix.
Note: while the fix was in Python, issue can surface in Java, PHP and Go also because they use the Python local dev server code.
I do believe that the cause of the problem is a bug in the local dev server (GoogleAppEngineLauncher), recently released.
I'm experiencing something similar in the PHP runtime: GloudStorage fails locally
I've been looking high and low. The microsoft website even has this link - https://www.windowsazure.com/en-us/develop/python/common-tasks/install-python/ but when I follow the instruction to execute the curl command and I type in my password it blows up with 'an error occured...' message.
I also found this - http://www.windowsazure.com/en-us/downloads/ - but when I click on the download link I always get a 400KB file (which feels small for an SDK) and if I try to untar it I get a bunch of errors.
Can anyone point me to a reliable mechanism for getting the SDK? I've gotta believe it's out there somewhere.
From python SDK Azure. In install python and the SDK you can see how can be installed.