Develop Reference

arch x86_64 and arm64e is available but python3 is saying incompatible architecture on Mac M1

I am trying to run this reading-text-in-the-wild
on Mac M1.
When I attempt to run this code
python3 make_keras_charnet_model.py
I get the error
Using Theano backend.
Traceback (most recent call last):
File "/Users/name/miniforge3/envs/ocr_env/lib/python3.8/site-packages/theano/gof/cutils.py", line 305, in <module>
from cutils_ext.cutils_ext import * # noqa
ImportError: dlopen(/Users/name/.theano/compiledir_macOS-12.0-arm64-i386-64bit-i386-3.8.6-64/cutils_ext/cutils_ext.so, 0x0002): tried: '/Users/name/.theano/compiledir_macOS-12.0-arm64-i386-64bit-i386-3.8.6-64/cutils_ext/cutils_ext.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/usr/local/lib/cutils_ext.so' (no such file), '/usr/lib/cutils_ext.so' (no such file)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/name/miniforge3/envs/ocr_env/lib/python3.8/site-packages/theano/gof/cutils.py", line 316, in <module>
from cutils_ext.cutils_ext import * # noqa
ImportError: dlopen(/Users/name/.theano/compiledir_macOS-12.0-arm64-i386-64bit-i386-3.8.6-64/cutils_ext/cutils_ext.so, 0x0002): tried: '/Users/name/.theano/compiledir_macOS-12.0-arm64-i386-64bit-i386-3.8.6-64/cutils_ext/cutils_ext.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/usr/local/lib/cutils_ext.so' (no such file), '/usr/lib/cutils_ext.so' (no such file)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "make_keras_charnet_model.py", line 10, in <module>
from keras.models import Sequential, model_from_json
File "/Users/name/miniforge3/envs/ocr_env/lib/python3.8/site-packages/keras/models.py", line 15, in <module>
from . import backend as K
File "/Users/name/miniforge3/envs/ocr_env/lib/python3.8/site-packages/keras/backend/__init__.py", line 47, in <module>
from .theano_backend import *
File "/Users/name/miniforge3/envs/ocr_env/lib/python3.8/site-packages/keras/backend/theano_backend.py", line 1, in <module>
import theano
File "/Users/name/miniforge3/envs/ocr_env/lib/python3.8/site-packages/theano/__init__.py", line 76, in <module>
from theano.scan_module import scan, map, reduce, foldl, foldr, clone
File "/Users/name/miniforge3/envs/ocr_env/lib/python3.8/site-packages/theano/scan_module/__init__.py", line 40, in <module>
from theano.scan_module import scan_opt
File "/Users/name/miniforge3/envs/ocr_env/lib/python3.8/site-packages/theano/scan_module/scan_opt.py", line 59, in <module>
from theano import tensor, scalar
File "/Users/name/miniforge3/envs/ocr_env/lib/python3.8/site-packages/theano/tensor/__init__.py", line 7, in <module>
from theano.tensor.subtensor import *
File "/Users/name/miniforge3/envs/ocr_env/lib/python3.8/site-packages/theano/tensor/subtensor.py", line 27, in <module>
import theano.gof.cutils # needed to import cutils_ext
File "/Users/name/miniforge3/envs/ocr_env/lib/python3.8/site-packages/theano/gof/cutils.py", line 319, in <module>
compile_cutils()
File "/Users/name/miniforge3/envs/ocr_env/lib/python3.8/site-packages/theano/gof/cutils.py", line 283, in compile_cutils
cmodule.GCC_compiler.compile_str('cutils_ext', code, location=loc,
File "/Users/name/miniforge3/envs/ocr_env/lib/python3.8/site-packages/theano/gof/cmodule.py", line 2212, in compile_str
return dlimport(lib_filename)
File "/Users/name/miniforge3/envs/ocr_env/lib/python3.8/site-packages/theano/gof/cmodule.py", line 299, in dlimport
rval = __import__(module_name, {}, {}, [module_name])
ImportError: dlopen(/Users/name/.theano/compiledir_macOS-12.0-arm64-i386-64bit-i386-3.8.6-64/cutils_ext/cutils_ext.so, 0x0002): tried: '/Users/name/.theano/compiledir_macOS-12.0-arm64-i386-64bit-i386-3.8.6-64/cutils_ext/cutils_ext.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/usr/local/lib/cutils_ext.so' (no such file), '/usr/lib/cutils_ext.so' (no such file)
I have duplicated my terminal to have the duplicate open with Rosetta and still I get the error.
when I run the command below to check the architecture available on my M1 Mac
file /bin/bash
I get this output
/bin/bash: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64e:Mach-O 64-bit executable arm64e]
/bin/bash (for architecture x86_64): Mach-O 64-bit executable x86_64
/bin/bash (for architecture arm64e): Mach-O 64-bit executable arm64e
I looks like I have both x86_64 and arm64e available but the error is saying
(mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/usr/local/lib/cutils_ext.so' (no such file), '/usr/lib/cutils_ext.so' (no such file)
what is causing this error and how can I fix it?

I've read A LOT of information about this problem and since I've installed python3-intel64 from official Python website I have no more troubles)
Solution:
Go to https://www.python.org/downloads/macos/
Find your version of Python for project
Press "Download macOS 64-bit universal2 installer"
Install Python from archive (since that moment you can use "python3-intel64" command to run Python instead of just "python")
Create new venv in project from base interpreter which you are installed
Install all requirements to venv
PROFIT!!!

the package you are using is not compatible with the new mac hardware by the looks of it. You will need to run this using Rosetta 2, which is an apple system component.

I found that i need to specify the architecture
so instead of
python3 make_keras_charnet_model.py
i now use this
arch -arm64 python3 make_keras_charnet_model.py

if you install Python 3.8 or 3.9 with the macOS 64-bit universal2 installer package. you may try using python3-intel64 instead of python3.
in my MacBook M1
/usr/local/bin/python3-intel64

While the virtual env is active, I usually reinstall the package that complains with
pip install -U --force <package name>

using Rosetta
arch -arm64 python3 -m pip install theano

Use the next command to install your libraries or tu run your script:
arch -x86_64 python3 -pip install [your lib]
I recommend to remove the previously installed packages to avoid the error

How do I know which .dll is missing in python with cx_freeze?

I have a project which I am packaging with cx_freeze. When I try to run the resulting exe I get an ImportError: DLL load failed: The specified module could not be found.
Here is my setup.py:
from cx_Freeze import setup, Executable
# Dependencies are automatically detected, but it might need
# fine tuning.
buildOptions = dict(build_exe='liquidation', packages=['scipy', 'numpy'], excludes=['scipy.spatial.cKDTree'],
includes=['scipy', 'numpy', 'numpy.core._methods', 'scipy.sparse.csgraph._validation',
'numpy.lib.format', 'numpy.linalg._umath_linalg', 'scipy.sparse._csparsetools',
'scipy.sparse.linalg.isolve._iterative', 'scipy.sparse.linalg.eigen.arpack._arpack',
'scipy.special._ufuncs_cxx', 'scipy.special.specfun', 'scipy.integrate._odepack',
'scipy.integrate._quadpack', 'scipy.integrate.vode', 'scipy.integrate._dop',
'scipy.integrate.lsoda', 'scipy.optimize._minpack', 'scipy.optimize._zeros',
'scipy.spatial', 'scipy.spatial.ckdtree', 'scipy.spatial.kdtree',
'scipy._distributor_init', 'numpy.core._multiarray_umath'])
base = 'Console'
executables = [
Executable('liquidation.py', base=base, targetName='liquidation.exe')
]
setup(name='liquidation',
version='2.0',
description='Program to run and test Liquidation algorithms',
options=dict(build_exe=buildOptions),
executables=executables)
When I run python setup.py build_exe it seems to complete successfully, but when I run the exe I get the following output:
Traceback (most recent call last):
File "C:\Users\tamar\PycharmProjects\liquidation\venv\lib\site-packages\numpy\core\__init__.py", line 24, in <module>
from . import multiarray
File "C:\Users\tamar\PycharmProjects\liquidation\venv\lib\site-packages\numpy\core\multiarray.py", line 14, in <module>
from . import overrides
File "C:\Users\tamar\PycharmProjects\liquidation\venv\lib\site-packages\numpy\core\overrides.py", line 7, in <module>
from numpy.core._multiarray_umath import (
ImportError: DLL load failed: The specified module could not be found.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\tamar\PycharmProjects\liquidation\venv\lib\site-packages\cx_Freeze\initscripts\__startup__.py", line 40, in run
module.run()
File "C:\Users\tamar\PycharmProjects\liquidation\venv\lib\site-packages\cx_Freeze\initscripts\Console.py", line 37, in run
exec(code, {'__name__': '__main__'})
File "liquidation.py", line 64, in <module>
from io_tools.stock_wrapper import StockWrapper
File "C:\Users\tamar\PycharmProjects\liquidation\io_tools\stock_wrapper.py", line 4, in <module>
from io_tools.io_helper import *
File "C:\Users\tamar\PycharmProjects\liquidation\io_tools\io_helper.py", line 7, in <module>
from numpy import float_
File "C:\Users\tamar\PycharmProjects\liquidation\venv\lib\site-packages\numpy\__init__.py", line 142, in <module>
from . import core
File "C:\Users\tamar\PycharmProjects\liquidation\venv\lib\site-packages\numpy\core\__init__.py", line 54, in <module>
raise ImportError(msg)
ImportError:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy c-extensions failed.
- Try uninstalling and reinstalling numpy.
- If you have already done that, then:
1. Check that you expected to use Python3.7 from "C:\Users\tamar\PycharmProjects\liquidation\liquidation\liquidation.exe",
and that you have no directories in your PATH or PYTHONPATH that can
interfere with the Python and numpy version "1.18.3" you're trying to use.
2. If (1) looks fine, you can open a new issue at
https://github.com/numpy/numpy/issues. Please include details on:
- how you installed Python
- how you installed numpy
- your operating system
- whether or not you have multiple versions of Python installed
- if you built from source, your compiler versions and ideally a build log
- If you're working with a numpy git repository, try `git clean -xdf`
(removes all files not under version control) and rebuild numpy.
Note: this error has many possible causes, so please don't comment on
an existing issue about this - open a new one instead.
Original error was: DLL load failed: The specified module could not be found.
I understand that I need to point cx_freeze to the missing DLL's (either by adding them to buildOptions.include_files or by actually copying the DLL's to the build folder) but I am at a loss as to how to know which DLL's I am missing. numpy.core._multiarray_umath is in the build folder so I don't know what it could be.
I believe I am using the latest versions of numpy (1.18.3), scipy (1.4.1) and cx-freeze (6.1). Any help would be much appreciated!
Update: I used Dependency Walker to try and see what I am missing and it listed a file by the name of LIBOPENBLAS.SVHFG5YE3RK3Z27NVFUDAPL2O3W6IMXW.GFORTRAN-WIN32.DLL which cannot be found. All of the other dependencies have been copied into the build so it makes sense that this would be the missing one. Does anyone know where I can find this file?

Python error in Vim when run inside a 3.5.3 virtualenv

I have multiple Python versions installed with pyenv on Ubuntu 16.04. I create virtual environments to work with projects using different versions with pipenv. However, for Python 3.5.x only I get errors like this when I try to edit a file with Vim 8 (installed via the PPA) in an activated virtualenv:
"test.py" 49L, 1434C
Error detected while processing function jedi#init_python[4]..<SNR>48_init_python[27]..jedi#setup_python_imports:
line 25:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/user/.local/share/virtualenvs/test-AYf6DmHL/lib/python3.5/imp.py", line 19, in <module>
from importlib._bootstrap import _ERR_MSG, _exec, _load, _builtin_from_name
File "/home/user/.local/share/virtualenvs/test-AYf6DmHL/lib/python3.5/importlib/__init__.py", line 57, in <module>
import types
File "/home/user/.local/share/virtualenvs/test-AYf6DmHL/lib/python3.5/types.py", line 166, in <module>
import functools as _functools
File "/home/user/.local/share/virtualenvs/test-AYf6DmHL/lib/python3.5/functools.py", line 23, in <module>
from weakref import WeakKeyDictionary
File "/home/user/.local/share/virtualenvs/test-AYf6DmHL/lib/python3.5/weakref.py", line 12, in <module>
from _weakref import (
ImportError: cannot import name '_remove_dead_weakref'
The issue seems to affect all python related Vim plugins. With other Python versions (e.g. 3.6.x, 3.7.x) this does not occur. What may be causing this issue?
Update: It seems that when I run Vim inside a 3.5.x virtualenv, the Vim's interpreter's sys.path gets updated somehow. For example, this is the output of :py3 import sys; print(sys.path) inside a 3.6.0 virtualenv:
['/usr/lib/python35.zip', '/usr/lib/python3.5', '/usr/lib/python3.5/plat-x86_64-linux-gnu', '/usr/lib/python3.5/lib-dynload', '/usr/local/lib/python3.5
/dist-packages', '/usr/lib/python3/dist-packages', '_vim_path_']
And this is for 3.5.3:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/testuser/.local/share/virtualenvs/venv-3.5-8feRiT5s/lib/python3.5/imp.py", line 19, in <module>
from importlib._bootstrap import _ERR_MSG, _exec, _load, _builtin_from_name
File "/home/testuser/.local/share/virtualenvs/venv-3.5-8feRiT5s/lib/python3.5/importlib/__init__.py", line 57, in <module>
import types
File "/home/testuser/.local/share/virtualenvs/venv-3.5-8feRiT5s/lib/python3.5/types.py", line 166, in <module>
import functools as _functools
File "/home/testuser/.local/share/virtualenvs/venv-3.5-8feRiT5s/lib/python3.5/functools.py", line 23, in <module>
from weakref import WeakKeyDictionary
File "/home/testuser/.local/share/virtualenvs/venv-3.5-8feRiT5s/lib/python3.5/weakref.py", line 12, in <module>
from _weakref import (
ImportError: cannot import name '_remove_dead_weakref'
['/must>not&exist', '/home/testuser/.local/share/virtualenvs/venv-3.5-8feRiT5s/lib/python35.zip', '/home/testuser/.local/share/virtualenvs/venv-3.5-8feRiT5s/li
b/python3.5', '/home/testuser/.local/share/virtualenvs/venv-3.5-8feRiT5s/lib/python3.5/plat-x86_64-linux-gnu', '/home/testuser/.local/share/virtualenvs/venv-3.
5-8feRiT5s/lib/python3.5/lib-dynload', '/home/testuser/.pyenv/versions/3.5.7/lib/python3.5', '/home/testuser/.local/share/virtualenvs/venv-3.5-8feRiT5s/lib/pyt
hon3.5/site-packages']
The issue can be reproduced in a ubuntu:16.04 docker container.

Debian Bug report logs - #852163: python3.5-minimal wont configure demonstrates that this error (for Python 3.5 and Python 3.6) is due to another incompatible installation of Python in /usr/local.
Some further digging with another user in #debian who was experiencing this
problem indicated that it was related to a locally installed
/usr/local/bin/python3.5. The stdlib in /usr/lib/python3.5 is picked up by the
/usr/local/bin/python3.5 but is not compatible with it, resulting in the
stacktrace shown.
Removing /usr/local/bin/python3* and all traces of that installation from
/usr/local/lib (plus other associated tools) solved this problem for the user
I was helping in #debian.
-- Stuart Prescott (link)
The problem is caused by the system installed python, which conflicts with personally installed python.

It could be the case that other python versions are interfering here. Do you have a PYTHONPATH set?
echo $PYTHONPATH
If it is set you could try to unset it. It's usually not needed when using virtual environments.

Try running python with sudo. I think the python which comes with OS needs to run with admin privilege.

ImportError: C extension: No module named 'parsing' not built

I have been trying to find a solution to this import error regarding the pandas library when it says no module named "parsing." Every library should be installed correctly from the interpreter and they are all the latest version.
This is what the console returns:
Traceback (most recent call last):
File "C:\Users\shaya\PycharmProjects\NEA\venv\lib\site-packages\pandas\__init__.py", line 26, in <module>
from pandas._libs import (hashtable as _hashtable,
File "C:\Users\shaya\PycharmProjects\NEA\venv\lib\site-packages\pandas\_libs\__init__.py", line 4, in <module>
from .tslib import iNaT, NaT, Timestamp, Timedelta, OutOfBoundsDatetime
File "pandas\_libs\tslibs\conversion.pxd", line 11, in init pandas._libs.tslib
File "pandas\_libs\tslibs\conversion.pyx", line 40, in init pandas._libs.tslibs.conversion
ModuleNotFoundError: No module named 'parsing'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:/Users/shaya/PycharmProjects/NEA/Main.py", line 4, in <module>
import pandas_datareader.data as data
File "C:\Users\shaya\PycharmProjects\NEA\venv\lib\site-packages\pandas_datareader\__init__.py", line 2, in <module>
from .data import (DataReader, Options, get_components_yahoo,
File "C:\Users\shaya\PycharmProjects\NEA\venv\lib\site-packages\pandas_datareader\data.py", line 7, in <module>
from pandas_datareader.av.forex import AVForexReader
File "C:\Users\shaya\PycharmProjects\NEA\venv\lib\site-packages\pandas_datareader\av\__init__.py", line 3, in <module>
from pandas_datareader.base import _BaseReader
File "C:\Users\shaya\PycharmProjects\NEA\venv\lib\site-packages\pandas_datareader\base.py", line 7, in <module>
import pandas.compat as compat
File "C:\Users\shaya\PycharmProjects\NEA\venv\lib\site-packages\pandas\__init__.py", line 35, in <module>
"the C extensions first.".format(module))
ImportError: C extension: No module named 'parsing' not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace --force' to build the C extensions first.
All of the above tracebacks are from PyCharm.
OS: Windows.
I am using pip to install packages
Python version: 3.7.1, panda version: 0.23.4

Do you have python added to path? To test this; open a cmd and type python. If it is on your path, you should see the version of python you are running (Assuming you are using a windows machine). If this is the case, you can simply run the command after you checked this. If not, please navigate to the location where Python is installed, open python.exe and try to run the command python setup.py build_ext --inplace --force
If this doesn't work, you should try to re-install pandas by pip install --upgrade --force-reinstall pandas
If this also fails you could also go rigourous, and simply create a new environment and install pandas there. Sidenote: It is probably better to install pandas by using conda package manager, Pandas has portions of its code written in C to make it run faster. If you tried to install pandas manually you would need to build it.

I had the same problem under the same circumstances. I went through the code of some of the pandas files and saw that there is indeed a module named 'parsing' in the tslib folder of my pandas directory, yet for some reason it's not able to call it. I just reinstalled python and now it's working for me. If you find any other alternative, please let me know.

Can't run python on virtualenv in PowerShell on Windows 10

Python and pip work just fine in PowerShell, but it's different if I run it while activating virtualenv. I got these errors:
(<project_name>-env) PS D:\Kerja\HIT\Python Projects\<project_name>\<project_name>-env\Scripts> python
Traceback (most recent call last):
File "D:\Kerja\HIT\Python Projects\<project_name>\<project_name>-env\lib\site.py", line 703, in <module>
main()
File "D:\Kerja\HIT\Python Projects\<project_name>\<project_name>-env\lib\site.py", line 692, in main
aliasmbcs()
File "D:\Kerja\HIT\Python Projects\<project_name>\<project_name>-env\lib\site.py", line 515, in aliasmbcs
import locale, codecs
File "D:\Kerja\HIT\Python Projects\<project_name>\<project_name>-env\lib\locale.py", line 17, in <module>
import functools
ImportError: No module named functools
(<project_name>-env) PS D:\Kerja\HIT\Python Projects\<project_name>\<project_name>-env\Scripts> pip
Fatal error in launcher: Unable to create process using '"'
How can I run python and pip successfully in virtualenv while using PowerShell?

I found the answer! It turns out when I use virtualenv on PowerShell, it prioritizes using python and pip from the virtualenv over the PATH environment variable.
The python and pip I mentioned before are located on /<environment_name/Scripts/> folder, along with activate and deactivate script. Deleting python.exe and pip.exe or just simply rename it will do.
Please note, it may affect pip freeze -l. I am not sure though.