I am trying to use python "requests" package in aws-glue. I tried with making .egg file for that package and it won't work. Also tried the making zip of request package and added to the glue job.
How can I execute below lines of code in AWS-glue?
import requests
print ("Hello")
Is it spark or python shell job? Answer depends on it.
For python shell job:
Use similar to following line in your setup.py file:
install_requires=['requests==2.22.0']
Replace requests with any library you wish to install.
Related
I need to run some long running job via Azure Web Job in Python.
I am facing below error trying to import pandas.
File "D:\local\Temp\jobs\triggered\demo2\eveazbwc.iyd\pandas_init_.py", line 13
missing_dependencies.append(f"{dependency}: {e}")
The Web app (under which I will run the web job) also has python code using pandas, and it does not throw any error.
I have tried uploading pandas and numpy folder inside the zip file (creating venv, installing packages and zipping Lib/site-packages content), (for 32 bit and 64 bit python) as well as tried appending 'D:/home/site/wwwroot/my_app_name/env/Lib/site-packages' to sys.path.
I am not facing such issues in importing standard python modules or additional package modules like requests.
Error is also thrown in trying to import numpy.
So, I am assuming some kind of version mismatch is happening somewhere.
Any pointers to solve this will be really useful.
I have been using Python 3.x, not sure if I should try Python 2.x (virtual env, install package and zip content of Lib/site-packages).
Regards
Kunal
The key to solving the problem is to add Python 3.6.4 x64 Extension on the portal.
Steps:
Add Extensions on portal.
Create a requirements.txt file.
pandas==1.1.4
numpy==1.19.3
Create run.cmd.
Create below file and zip them into a single zip.
Upload the zip for the webjob.
For more details, please read this post.
Webjobs Running Error (3587fd: ERR ) from zipfile
I am currently interested in installing the following git repository:
https://github.com/CodeReclaimers/neat-python/tree/master/examples/xor.
It contains a file called visualize.py and I would love to just install and use it as a module (e.g. numpy). However, I'm not sure how if it is possible to do this and was, therefore, hoping anyone could clarify this for me.
I have tried:
pip install git+https://github.com/CodeReclaimers/neat-python/tree/master/examples/xor
Any help would be appreciated!
Edit:
I was able to clone the entire repo:
pip install git+https://github.com/CodeReclaimers/neat-python.git
Does this mean I should be able to use all the files available in this repository as a module or is there something I'm still missing? I still cannot use visualize as a module. Thanks!
If you are able to clone the library to the same directory, you can simply import the python file without pip installing as module.
For instance if the file is called myFile.py, you can call the following and the entire file is executed and functions within can be used.
import myFile
1- open directory with modules
example : c:\users\james\appdata\local\programs\python\python39\lib
2-New python file created
example : visualize.py
3- save the python file.
How am I supposed to view the source code of a third party python module (pywhatkit) that I installed (using pip).
I also tried using the open() and read() functions, but it didn't work and the output shows.
Err: file not found
Python library source code often is stored in github.
In this case the source code you want you can find here:
https://github.com/Ankit404butfound/awesomepy/blob/master/program.py
If you are using an IDE like Pycharm, you can ctrl-click on that module name in your code and it will take you to the file with the definition. Normally, the installed modules are found in lib/site-packages/ of your python installation directory.
I'm using netbeans to write a simple python program which I need the requests module for, I've downloaded requests through terminal and it all seems to be fine there but netbeans can't seem to find it.
This is the error that it's throwing up:
import requests
ImportError: No module named requests
I've tried installing the requests library directly into the python folder but the folder won't let me paste anything into it.
There do seem to be answers on the netbeans forums but their server is down so won't let me on their website to my annoyance!
EDIT
I've tried to run python setup.py install as per other answers on the website but had no luck.
EDIT
have tried completely uninstalling python and requests to make sure it wasn't an installation error but still no luck.
This clearly looks like an error of installation of the request module to some other place than where your netbeans expects when running the code.
In your console run
which python
Check if this gives the same path as the one set in your netbeans. You can set your path by adding new platform using Tools > Python Platforms > New:
I would suggest that you learn bit more about sandboxed environments such as virtualenv. This article shows how you can use a virtualenv to install packages and use the same virtualenv for netbeans so that whatever packages you install in the virtualenv will be available in the netbeans for you to use. For this case, it could be requests.
In the end I gave up with requests, as I was using requests to get json data from an API I decided just to go back to the drawing board and start over rather than attempt to fix something that I couldn't work out. I am now using the urllib import and whilst this may not be the most efficient way it works which is the most important thing.
I am building a Python script for processing json data according to some criteria.
Also I have built a custom module which consists of methods for retrieving json data and generation of json file which consist of processed data.
But that module file is stored into S3 bucket and I need to import that module into my script so that I can invoke functions defined in module.
Please suggest me appropriate solution regarding importing python module from external URL
Well, you could download the file using urllib2 and then import it, if the online module is all in one file:
from urllib2 import urlopen
r = urlopen('http://urlHere/fileHere')
f = open('filenameToWrite', 'w')
f.write(r.read())
f.close()
import filenameWithout.PyInIt
Package your module into your favorite extension (tarball, wheel, etc.) using setuptools and then you will be able to install it using pip as bruno by doing something like:
pip install --no-index --trusted-host s3_ip/host --find-links http://s3.com...
Please see my answer to question # 48905127 - believed to be the best solution I found so far and comes with detailed steps and code snippets ready for you to copy and use.