Develop Reference

[Windows 10 / Python 3.9 / Pipenv]How to debug an error when using `pipenv install xxx` from command line

I'm having trouble installing things without error when using Pipenv and am unable to figure out how to fix the error. Just for some upfront clarification, I'm using Windows 10, Python 3.9 and the issue is when I install with Pipenv. My last name is a little odd and has an apostrophe in it; however, I usually leave the apostrophe out when registering for things due to potential issues with the apostrophe. For some reason, at work, the IT department created my account using the apostrophe. Since it is configured with Azure AD, I cannot adjust the username from my own PC, even though I have admin rights on this machine.
The error seems to be coming from my opening a JSON file in my appdata directory, which , when fully qualified contains the string "t'alua" (my last name). Unfortunately, the string seems to be enclosed in single quotes so the apostrophe in my name causes a parsing issue. Just for the record, I have even uninstalled python from my C drive and installed on my D drive and executed commands in my D drive (where there is no apostrophe in the working directory's path) but I still get the same issue since it is related to the appdata directory's absolute path. I have also asked the IT department to change my username to remove the apostrophe but they said its not feasible due to so many downstream changes that would be required to prevent any kind of bugs, which I understand.
Here is an example command with the error shown:
(corl_anl_coercion-sJyNKu6O) D:\mttf_mtbf_datasets\training_data\corl_anl_coercion>pipenv install numpy
Installing numpy...
[= ] Installing numpy...Failed to load paths: File "<string>", line 1
import sysconfig, distutils.sysconfig, io, json, sys; paths = {u'purelib': u'{0}'.format(distutils.sysconfig.get_python_lib(plat_specific=0)),u'stdlib': u'{0}'.format(sysconfig.get_path('stdlib')),u'platlib': u'{0}'.format(distutils.sysconfig.get_python_lib(plat_specific=1)),u'platstdlib': u'{0}'.format(sysconfig.get_path('platstdlib')),u'include': u'{0}'.format(distutils.sysconfig.get_python_inc(plat_specific=0)),u'platinclude': u'{0}'.format(distutils.sysconfig.get_python_inc(plat_specific=1)),u'scripts': u'{0}'.format(sysconfig.get_path('scripts')),u'py_version_short': u'{0}'.format(distutils.sysconfig.get_python_version()), }; value = u'{0}'.format(json.dumps(paths));fh = io.open('c:/users/t'alua~1/appdata/local/temp/tmp0m5ovbrb.json', 'w'); fh.write(value); fh.close()
^
SyntaxError: invalid syntax
Output:
[ ==] Installing numpy...Failed to load paths: File "<string>", line 1
import sysconfig, distutils.sysconfig, io, json, sys; paths = {u'purelib': u'{0}'.format(distutils.sysconfig.get_python_lib(plat_specific=0)),u'stdlib': u'{0}'.format(sysconfig.get_path('stdlib')),u'platlib': u'{0}'.format(distutils.sysconfig.get_python_lib(plat_specific=1)),u'platstdlib': u'{0}'.format(sysconfig.get_path('platstdlib')),u'include': u'{0}'.format(distutils.sysconfig.get_python_inc(plat_specific=0)),u'platinclude': u'{0}'.format(distutils.sysconfig.get_python_inc(plat_specific=1)),u'scripts': u'{0}'.format(sysconfig.get_path('scripts')),u'py_version_short': u'{0}'.format(distutils.sysconfig.get_python_version()), }; value = u'{0}'.format(json.dumps(paths));fh = io.open('c:/users/t'alua~1/appdata/local/temp/tmpywegc7gn.json', 'w'); fh.write(value); fh.close()
^
SyntaxError: invalid syntax
Output:
Adding numpy to Pipfile's [packages]...
Installation Succeeded
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Resolving dependencies...
Locking Failed!
Traceback (most recent call last):
File "D:\installations\python\lib\site-packages\pipenv\resolver.py", line 764, in <module>
main()
File "D:\installations\python\lib\site-packages\pipenv\resolver.py", line 758, in main
_main(parsed.pre, parsed.clear, parsed.verbose, parsed.system, parsed.write,
File "D:\installations\python\lib\site-packages\pipenv\resolver.py", line 741, in _main
resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
File "D:\installations\python\lib\site-packages\pipenv\resolver.py", line 702, in resolve_packages
results, resolver = resolve(
File "D:\installations\python\lib\site-packages\pipenv\resolver.py", line 684, in resolve
return resolve_deps(
File "d:\installations\python\lib\site-packages\pipenv\utils.py", line 1406, in resolve_deps
results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
File "d:\installations\python\lib\site-packages\pipenv\utils.py", line 1119, in actually_resolve_deps
resolver.resolve()
File "d:\installations\python\lib\site-packages\pipenv\utils.py", line 834, in resolve
results = self.resolver.resolve(max_rounds=environments.PIPENV_MAX_ROUNDS)
File "d:\installations\python\lib\site-packages\pipenv\utils.py", line 822, in resolver
self.get_resolver(clear=self.clear, pre=self.pre)
File "d:\installations\python\lib\site-packages\pipenv\utils.py", line 813, in get_resolver
constraints=self.parsed_constraints, repository=self.repository,
File "d:\installations\python\lib\site-packages\pipenv\utils.py", line 806, in parsed_constraints
self._parsed_constraints = [c for c in self.constraints]
File "d:\installations\python\lib\site-packages\pipenv\utils.py", line 806, in <listcomp>
self._parsed_constraints = [c for c in self.constraints]
File "d:\installations\python\lib\site-packages\pipenv\patched\notpip\_internal\req\req_file.py", line 137, in parse_requirements
for parsed_line in parser.parse(filename, constraint):
File "d:\installations\python\lib\site-packages\pipenv\patched\notpip\_internal\req\req_file.py", line 282, in parse
for line in self._parse_and_recurse(filename, constraint):
File "d:\installations\python\lib\site-packages\pipenv\patched\notpip\_internal\req\req_file.py", line 287, in _parse_and_recurse
for line in self._parse_file(filename, constraint):
File "d:\installations\python\lib\site-packages\pipenv\patched\notpip\_internal\req\req_file.py", line 329, in _parse_file
args_str, opts = self._line_parser(line)
File "d:\installations\python\lib\site-packages\pipenv\patched\notpip\_internal\req\req_file.py", line 365, in parse_line
shlex.split(options_str), defaults) # type: ignore
File "D:\installations\python\lib\shlex.py", line 315, in split
return list(lex)
File "D:\installations\python\lib\shlex.py", line 300, in __next__
token = self.get_token()
File "D:\installations\python\lib\shlex.py", line 109, in get_token
raw = self.read_token()
File "D:\installations\python\lib\shlex.py", line 191, in read_token
raise ValueError("No closing quotation")
ValueError: No closing quotation
I can see from the output above that the error is due to this line of python being executed:
fh = io.open('c:/users/t'alua~1/appdata/local/temp/tmp0m5ovbrb.json', 'w');
since the string is not terminated correctly.
Unfortunately, I'm not sure where this is coming from, though. I tried to read through each of the python files listed in the stack trace and I see where, in shlex.py if the string ends, while still being in a 'quoted state', then it raises the "No closing Quotation" valueerror. I couldn't see any line of code where it actually pulls the appdata path or why it is using appdata in the first place. I did find in my python installation's site packages directory's util.py:
D:\installations\python\Lib\site-packages\pipenv\utils.py
that there is an escaped_grouped_arguments function. I thought perhaps if the filepath is being parsed with this, I can modify the function to check for variations of my lastname and replace it with a good value so I made the following change:
def escape_grouped_arguments(s):
"""Prepares a string for the shell (on Windows too!)
Only for use on grouped arguments (passed as a string to Popen)
"""
if s is None:
return None
## This is the original code
# Additional escaping for windows paths
#if os.name == "nt":
# s = "{}".format(s.replace("\\", "\\\\"))
#return '"' + s.replace("'", "'\\''") + '"'
## This is my modification
if "t'alua" in s.lower():
bad_names = ["T'Alua","t'alua","T'alua","T'ALUA"]
good_name = "talua"
for bad_name in bad_names:
s = s.replace(bad_name, good_name)
if os.name == "nt":
s = "{}".format(s.replace("\\", "\\\\"))
return '"' + s.replace("'", "'\\''") + '"'
But this change didn't seem to resolve the errors. Any and all guidance on how to correctly spot exactly where the error is coming from as well as how to best fix this particular issue would be greatly appreciated.

Pandas can't find the relevant file

I'm attempting to try a t-test in python using the pandas module. However, the same error keeps occuring in which my target file cannot be found. In this case, the target file is brain_size.csv, where the separators are semi-colons. The values which are left blank are represented by a period.
Here's what I have keyed in:
import pandas as pd
data = pd.read_csv('This PC\Desktop\brain_size.csv', sep=';', na_values='.')
Here's the error message. It's a long string
File "C:\Users\Tina Gnali\Anaconda3\lib\site-packages\pandas\io\parsers.py", line 562, in parser_f
return _read(filepath_or_buffer, kwds)
File "C:\Users\Tina Gnali\Anaconda3\lib\site-packages\pandas\io\parsers.py", line 315, in _read
parser = TextFileReader(filepath_or_buffer, **kwds)
File "C:\Users\Tina Gnali\Anaconda3\lib\site-packages\pandas\io\parsers.py", line 645, in __init__
self._make_engine(self.engine)
File "C:\Users\Tina Gnali\Anaconda3\lib\site-packages\pandas\io\parsers.py", line 799, in _make_engine
self._engine = CParserWrapper(self.f, **self.options)
File "C:\Users\Tina Gnali\Anaconda3\lib\site-packages\pandas\io\parsers.py", line 1213, in __init__
self._reader = _parser.TextReader(src, **kwds)
File "pandas\parser.pyx", line 358, in pandas.parser.TextReader.__cinit__ (pandas\parser.c:3427)
File "pandas\parser.pyx", line 628, in pandas.parser.TextReader._setup_parser_source (pandas\parser.c:6861)
OSError: File b'This PC\\Desktop\x08rain_size.csv' does not exist
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
I want to ask:
What am I doing wrong? Why I can't retrieve the target file?
Why my error elicits such a long error message?
What does the "parser" module do?

The problem is with using backslashes "\". You must avoid that. Backslash is reserved for something called escape characters, like new line being denoted with "\n" and stuff. Either use double backslashes "\\" or just forwardslashes "/" or raw literals in your read_csv():
"C:\\Users\\blabla\\"
or
"C:/Users/blabla/"
or
r"C:\Users\blabla\"
Regarding how to identify the error, look for the "error" string in the error message. It is here:
OSError: File b'This PC\\Desktop\x08rain_size.csv' does not exist
This tells you that Python is looking for a file called 'x08rain_size.csv', and obviously you don't have such a file. But what is x08rain? Could it be that b is replaced with x08 when you place a backslash in front of it? Let's ask this to Python:
In [247]: '\b'
Out[247]: '\x08'
There we go!

Sometimes you may not be able to use
"C:\\Users\\blabla\\" or "C:/Users/blabla/"
Solution 1. The other option could be this:
Open Anaconda prompt or cmd, then change the path. Lets assume you are in drive "c" and your folder is in drive e. So after opening the cmd write "e:" and hit enter. Then the command will show you "E:\>". Now you should write "cd E:\Users\blabla\desired_folder". After running that, you should write "jupyter notebook" and run it. It will generate and open a new notebook in the same folder that has your file.
Solution 2. The other simple solution is, after opening the jupyther notebook> file> use the folder icon and choose the right folder.

May be the sep is different it may be "," try it and if still it doesnt works try removing sep and na values and try to keep the file in same directory where the program is present or give actual path

PyXB - AssertionError: No element bindings in http://www.w3.org/1999/xhtml

I am attempting to generate bindings for a WSDL with PyXB, and it is giving the AssertionError exception in the title.
My understanding, based on the PyXB documentation, is that the bundle archive for http://www.w3.org/1999/xhtml is included with PyXB. However, something appears to be wrong. It either does not get used, or it has incorrect contents.
I use the following command line to attempt to generate the bindings:
python c:\Python27\Scripts\pyxbgen.py --wsdl-location=http://xx.xxx.xxx.xxx/YYY.asmx?WSDL --module=client --write-for-customization
The traceback:
Traceback (most recent call last):
File "c:\Python27\Scripts\pyxbgen.py", line 51, in <module> generator.resolveExternalSchema()
File "c:\Python27\lib\site-packages\pyxb\binding\generate.py", line 2647, in resolveExternalSchema
schema = converter(self, sl)
File "c:\Python27\Scripts\pyxbgen.py", line 28, in WSDLToSchema
spec = wsdl.definitions.createFromDOM(pyxb.utils.domutils.StringToDOM(xmld,
location_base=wsdl_uri), process_schema=True, generation_uid=generator.generationUID())
File "c:\Python27\lib\site-packages\pyxb\binding\basis.py", line 1767, in createFromDOM
return self._createFromDOM(node, expanded_name, **kw)
File "c:\Python27\lib\site-packages\pyxb\binding\basis.py", line 1791, in _createFromDOM
return element.CreateDOMBinding(node, self.elementForName(expanded_name), **kw)
File "c:\Python27\lib\site-packages\pyxb\binding\basis.py", line 1735, in elementForName
assert 'elementBinding' in elt_en.namespace()._categoryMap(), 'No element bindings in %s' % (elt_en.namespace(),)
AssertionError: No element bindings in http://www.w3.org/1999/xhtml
In addition, I set the PYXB_ARCHIVE_PATH environment variable to:
C:\Python27\Lib\site-packages\pyxb\bundles\common\raw
I am not sure if this is the correct way to do this. I also tried specifying the --archive-path command line option as well, but I got the same error back.

Probably you need to use:
--archive-path=${PYXB_ROOT}/pyxb/bundles/common//:+
as the argument. This recursively searches for available namespaces in the common bundles first, then includes any other search paths. There's an example in the manual that's close to this.

ftp directory in Python 3

I have the following code in Python 3.3:
ftpHost="myhost"
link=ftplib.FTP(ftpHost)
link.login(user="me",passwd="alsome")
t=link.nlst()
link.quit()
print(t)
The output I get is
['.','..','dir1','dir2']
In the Python 3 document, it is written that FTP.nlst has been "deprecated since version 3.3: use mlsd() instead". When I tried
ftpHost="myhost"
link=ftplib.FTP(ftpHost)
link.login(user="me",passwd="alsome")
t=link.mlsd()
link.quit()
print(t)
The output I get is
<generator object mlsd at 0x0000000002A0F120>
which is not the desired answer. I can't figure out where within t can I get the directory listing. How can I use mlsd to get the directory listing?
If I try #fourtheye 's suggestion and change to
t=list(link.mlsd)
the error I get is
Traceback (most recent call last):
File "C:/xyz/python-test/python-test.py", line 17, in <module>
main()
File "C:/xyz/python-test/python-test.py", line 12, in main
t=list(link.mlsd())
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\ftplib.py", line 556, in mlsd
self.retrlines(cmd, lines.append)
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\ftplib.py", line 446, in retrlines
with self.transfercmd(cmd) as conn, \
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\ftplib.py", line 386, in transfercmd
return self.ntransfercmd(cmd, rest)[0]
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\ftplib.py", line 352, in ntransfercmd
resp = self.sendcmd(cmd)
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\ftplib.py", line 259, in sendcmd
return self.getresp()
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\ftplib.py", line 233, in getresp
raise error_perm(resp)
ftplib.error_perm: 500 MLSD not understood

Generators dont keep entire results in the memory. The results will be generated when asked for one, with next function. In order to get the values as a list, use the list function with link.mlsd(), which will internally call next on the generator object.
Just change
t=link.mlsd()
to
t=list(link.mlsd())

Who runs the FTP host? The 500 MLSD not understood means they don't support the MLSD command. That's a later extension to the FTP command set, so they're just out of date. The specification is here:
https://www.rfc-editor.org/rfc/rfc3659
Unless and until they upgrade to a "modern" version of FTP, just continue to use nlst().

aifc module Error: Python won't open file

I'm just getting started with python so please bear with me ;)
While following a basic tutorial I ran into a problem while opening a file, here's the traceback:
File "/home/nick/Dropbox/workspace/pytest/schlange.py", line 55, in <module>
f=open("file.csv","r")
File "/usr/lib/python2.6/aifc.py", line 922, in open
return Aifc_read(f)
File "/usr/lib/python2.6/aifc.py", line 335, in __init__
self.initfp(f)
File "/usr/lib/python2.6/aifc.py", line 288, in initfp
raise Error, 'file does not start with FORM id'
aifc.Error: file does not start with FORM id
Does that mean that 'aifc.py' in my python installation is broken or did I miss something important here?

You've decided to import * from aifc, and its open() has shadowed the built-in open(). This is why we don't import *. Import the module itself instead and use the reference to get to its names when required, e.g. aifc.open().