Develop Reference

submit job with pandas in a zip file

I have two libraries: Pandas and utils (my library), and I want to import in my code. Since I was testing Pandas does not work as well.
Using boto3 and requests (without being preinstalled in the cluster) it works creating two zip files:
libs.zip: with boto3 and requests
dependencies.zip: utils
So, I import Pandas using a requirements file and creating a zip with all Pandas dependencies. I've tried importing the zip file within the code, like:
sc.addPyFile("libs.zip")
and the spark submit is like:
spark-submit --deploy-mode client --py-files s3://${BUCKET_NAME}/libs.zip s3://${BUCKET_NAME}/main.py
I tried a lot to submit a spark job in EMR cluster and I don't have any idea about this issue:
Traceback (most recent call last):
File "/mnt/tmp/spark-xxxx/main.py", line 20, in <module>
import pandas as pd
File "/mnt/tmp/spark-xxxx/userFiles-xxxx/libs.zip/pandas/__init__.py", line 17, in <module>
ImportError: Unable to import required dependencies:
numpy:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.
We have compiled some common reasons and troubleshooting tips at:
https://numpy.org/devdocs/user/troubleshooting-importerror.html
Please note and check the following:
* The Python version is: Python3.7 from "/usr/bin/python3"
* The NumPy version is: "1.19.4"
and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.
Original error was: No module named 'numpy.core._multiarray_umath'
How can I import Pandas and another library (created by me) in spark submit.

How to import numpy through xlwings package? "ImportError: DLL load failed: The specified module could not be found."

I'm trying to use the 'Run Python' function of xlwings to run Python code through VBA.
I have been using Spyder to execute my code and it runs with no errors.
When trying to run this from VBA with the xlwings package I receive:
"ImportError: DLL load failed: The specified module could not be found."
and this error relates to the numpy package.
I tried uninstalling and reinstalling the anaconda package and using pip install numpy.
I checked that I have the most up to date version of xlwings 0.15.8.
I found this thread https://github.com/xlwings/xlwings/issues/954 stating this issue was fixed with version 0.15.7 of xlwings.
VBA code:
RunPython ("import Demand; Demand.calibrate_Demand()")
Spyder code:
import numpy as np
import xlwings as xw
import pandas as pd
import statsmodels.api as sm
from statsmodels.tsa.arima_model import ARMA
from statsmodels.tsa.arima_model import ARMAResults
from matplotlib import pyplot as plt
import datetime
def calibrate_dDemand():
My Python file is called Demand.py
When executing my VBA code I receive the following error:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "e:\julia\calibration automation\Demand.py", line 17, in <module>
import numpy as np
File "C:\Users\julia\AppData\Local\Continuum\anaconda3\lib\site-packages\numpy\__init__.py", line 140, in <module>
from . import _distributor_init
File "C:\Users\julia\AppData\Local\Continuum\anaconda3\lib\site-packages\numpy\_distributor_init.py", line 34, in <module>
from . import _mklinit
ImportError: DLL load failed: The specified module could not be found.
If I place import pandas as pd first (before importing numpy) I receive this error
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "e:\julia\calibration automation\Demand.py", line 19, in <module>
import pandas as pd
File "C:\Users\julia\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\__init__.py", line 19, in <module>
"Missing required dependencies {0}".format(missing_dependencies))
ImportError: Missing required dependencies ['numpy']

I've been trying for a couple of weeks to hit back against this problem too.
I have a company-owned laptop which is severely restricted to what environment variables I have access too, and PATH is not one of them.
I have no trouble linking Excel with Python (Spyder / Anaconda3 installation), but as soon as add import numpy as np to my python code, I get the same DLL load failed error. (It didn't always do this though, it just started doing it overnight for no reason I can fathom, but I digress). The code itself works fine in Spyder though.
I did eventually find something of a workaround though, but it's a bit of a last resort, being more than a bit awkward and daft. Maybe useful for a developer to track down a more sensible solution though?
Anyway, I found that if I launched Excel from Spyder, the numpy module correctly imports, and remains stable. To do this, I just access the contents of one of the workbook's cells without having Excel already open. This launches Excel, opens the workbooks, and correctly imports numpy.
What do you think?

AttributeError: 'module' object has no attribute 'DataFrame'

I am running Python 2.7.10 on a Macbook.
I have installed:
Homebrew
Python 2.x, 3.x
NI-VISA
pip
pyvisa, pyserial, numpy
PyVISA
Anaconda
Pandas
I am attempting to run this script. A portion of it can be read here:
import visa
import time
import panda
import sys
import os
import numpy
os.system('cls' if os.name == 'nt' else 'clear') #clear screen
rm = visa.ResourceManager()
rm.list_resources()
print(rm.list_resources())
results = panda.DataFrame(columns=['CURR', 'VOLT', 'TIME'])
This is what is returned on the command line, below.
Note the line that says
AttributeError: 'module' object has no attribute 'DataFrame'
(u'USB0::0x05E6::0x2280::4068201::INSTR', u'ASRL1::INSTR', u'ASRL2::INSTR', u'ASRL4::INSTR')
Traceback (most recent call last):
File "k2280.py", line 14, in <module>
results = panda.DataFrame(columns=['CURR', 'VOLT', 'TIME'])
AttributeError: 'module' object has no attribute 'DataFrame'
Any help or insight on this issue would be appreciated.

It's pandas, not panda, so use import pandas instead. It's also common practice to import pandas as pd for convenience:
import pandas as pd
df = pd.DataFrame()

The module is called pandas not panda
python3 -m pip install pandas
import pandas as pd
pd.DataFrame()

Please read if you're new to python.
As I am also new to python since 2 days and going through the tutorial.
What I know is pandas are the packages we install in the python library to the machine we are using.
As I am new and I was practicing to import and use pandas.DataFrame I kept my filename as pandas.py
And here is the error I was doing. I can't use pandas.py because the machine is assuming its module inside this pandas.py
I changed the filename and it start working fine.
Few things to know if you're getting errors.
You are using pandas.py as filename, you need to change the file name
You are not importing pandas in the file and started working on its module
You are not using DataFrame in camel case
I think these 3 things should be kept in mind to use DataFrame to avoid this error.

How do I fix: ImportError: cannot import name 'PandaRequest'?

This is my first time using a jupyter notebook.
I was trying to import the panda module:
import panda as pd
but I get the following error:
/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/panda/__init__.py in <module>()
----> 1 from request import PandaRequest
ImportError: cannot import name 'PandaRequest'
How to fix this?
Edit: I wanted to use pandas but typed panda instead. :'D

You have installed and imported the panda module, which is the a Python implementation of the Panda REST interface and which has incomplete dependencies.
What you probably want to do is install pandas (with the final s) and import that.

I got the same problem and come up with the following solutions:
import pandas as pd
instead of
import panda as pd
It is called pandas.

Execute pip install panda. You don't have the Panda module.
Edit: My mistake, it looks like there's an error within the Panda module itself. Does it rely on PandaRequest?

ImportError: cannot import name DataFrame

I am using Pandas for the first time and trying to import Pandas, DataFrame and pandas.io.data as such
import pandas as pd
from pandas import DataFrame
import pandas.io.data
I get the error
"ImportError: cannot import name DataFrame"
and
"ImportError: No module named io.data"
I have tried the following to find the source of error on Ubuntu 14.04:
$pip show pandas
-> Version: 0.14.0
Location: /home/new-pc/anaconda/lib/python2.7/site-packages
$$PATH
-> bash: /home/new-pc/anaconda/bin
Not sure where else to look for errors. Thanks for all the inputs.

You must be mixing Python installations, with several python programs/distributions (Anaconda), and maybe python version (2 and 3)
The following commands should not produce any error:
sudo apt-get install python
sudo pip install pandas
python -c'import pandas as pd; from pandas import DataFrame; import pandas.io.data'

It will happen if you call your file pandas.py and try to run it. You should rename the file then.
It will also occur if you save a pandas.py file in the Python directory. Check if you have any files called pandas.py there and rename/move those files to another location.

Try:
from pandas_datareader import data
as shown in the example here

Check if you have saved any file with same names as in pandas libraries, such as saving file name as DataFrame.py. Always avoid naming the program as same as library name check all the location.Once you remove/rename your python script will run without any issue.