I'm working with shelve in python 2.7.6 for caching calculations, and I've ran into the issue described HERE for shelve files produced by me and implemented the suggested solution in a function that merges a file other into the file target:
... # target and other are strings
# Loads the gdbm module, as suggested
mod = __import__("gdbm")
# Open target for modifications
tar = shelve.Shelf(mod.open(target, 'c', 0666)) # THROWS EXCEPTION
# Open other for reading
oth = shelve.Shelf(mod.open(other, 'r'))
...
The two files are owned by me, are recorded in a local filesystem and have posix permissions set to 0666, or, equivalently, -rw-rw-rw- in my Linux Mint box, so that the obvious checks have been performed:
$ ls -l
-rw-rw-rw- 1 myusr mygrp 11694080 Sep 17 21:24 cache
-rw-rw-rw- 1 myusr mygrp 12189696 Sep 17 21:23 cache.0
Here, cache is the target, and cache.0 is the other file. The current working directory is owned by me and has permissions 0775, and I can create files with touch, cp, etc., at will with no problems, and I've even set my umask to 0000, so new files are created with 0666, or, equivalently, -rw-rw-rw- permissions.
I've even matched the actual file permissions with the permissions in the gdbm.open() call, according to its documentation; however, to no avail.
Update: By running the python code with sudo, i.e., with superuser privileges, the error occurs on the same line; however with a different message: gdbm error: Bad magic number! This is very strange as the very point of using a (seemingly) lower-level module (gdbm as opposed to shelve) was precisely bypassing database type detection.
Update #2: Running python whichdb.py on the files returns dbhash; however, changing the module to the dbhash on the loading code still gives the following errors:
bsddb.db.DBAccessError: (13, 'Permission denied')
when running as user, but
bsddb.db.DBInvalidArgError: (22, 'Invalid argument -- BDB0210 ././merge-cache.py: metadata page checksum error')
when running with sudo; merge-cache.py is my code's name.
This new error is discussed here, in connection with python version, but (i) my python version is different than the one in that post, and (ii) files are created and later read with the same version of python.
This answer indicates shelve gets "wasted" with large sets, but the problem I'm reporting occurs with smaller databases as well.
Question: How can I open these shelve files using python-2.7.6? (upgrading python is not an option).
Related
We changed over our SQL server 2017 with Machine learning Services ( MLS ) from running R 3.3.3 and Python 3.5.2 - to R 3.5.2 and Python 3.7.1.
SQL 2017 CU22 installs the higher value of R & Python, so you then have to run an exe to tell SQL to use the higher R & Python versions that were installed on disk via CU22.
After this was done successfully, when we went to restart Python we got the error :
Msg 39021, Level 16, State 1, Line 0
Unable to launch runtime for 'Python' script. Please check the configuration of the 'Python' runtime.
Msg 39019, Level 16, State 2, Line 0
An external script error occurred:
Unable to launch the runtime. ErrorCode 0x80070057: 87(The parameter is incorrect.).
This is a problem description and placeholder for the solution below.
See below for a solution that worked successfully .
If you look inside
E:\Program Files\Microsoft SQL Server\MSSQL14.< instance >\MSSQL\Log\ExtensibilityLog in the pythonlauncher.lor or rlauncher.log files, you may see a line that looks similar to this :
[0000293C][00002B9C][Error] Session creation failed with: F
This is I think is the attempt by SQL 2017 MLS to write to disk.
The fix that worked for us :
(1) Create a new directory e.g. D:\MLSTEMP
(2) Create subdirectories D:\MLSTEMP\ < SQL_instance >00 and D:\MLSTEMP\ < SQL_instance >01
** Make sure you create all 21 directories < SQL_instance >00 to < SQL_instance >20 (inclusive) - or you will be limited to only one user at a time being able to run R / Python code on a box. **
(3) Make sure the local MSSQLLaunchpad service, Everyone group & SQL service account has full access to both the D:\MLSTEMP and the sub dirs.
Note : the < > brackets are not part of the directory name
Note: launchpad service name is : NTService\MSSQLLaunchpad$ < instance_name >
(4) Go to your
D:\Program Files\Microsoft SQL Server\MSSQL14.SQL001\MSSQL\Binn\pythonlauncher.config
and
D:\Program Files\Microsoft SQL Server\MSSQL14.SQL001\MSSQL\Binn\rlauncher.config
files,
and alter the WORKING_DIRECTORY setting in each file to be WORKING_DIRECTORY=D:\MLSTEMP
(5) Restart the MSSQLLaunchpad service on the box.
Interestingly, the ****01 directory is the one that's used, not the ****00 dir.
To get more detailed info in the rlauncher or pythonlauncher log files, in the rlauncher or pythonlauncher.config files, change the trace_level setting to 4 i.e. TRACE_LEVEL=4. Its normally set to 1.
It should now work. You may have to play with permissions a bit.
HTH someone....
It's a permission issue.
Go to: ...\Microsoft SQL Server\MSSQL13.MSSQLSERVER\MSSQL\Log\ExtensibilityLog folder and open pythonlauncher.log.
(note: if you have MSSQL13 and MSSQL14 folders: check both!)
At the end of the file there is a record similar to: [Error] Session creation failed with: Failed with 2 to get security for C:\ProgramData\MSSQLSERVER\Temp-PY\MSSQLSERVER01
this indicates an issue with permissions.
Go to: C:\ProgramData\MSSQLSERVER and grant everyone permissons on the Temp-PY shortcut (might not be necessarly everyone, but I'm not sure which exact user)
Restart SQL Server Launchpad service.
It worked for me :)
My code:
if os.path.exists(myDir):
shutil.rmtree(myDir)
os.mkdir(myDir)
Problem: It always work if myDir does not exist. If myDir exists, sometimes it throws error, sometimes it works.
Error log:
os.mkdir(myDir)
PermissionError: [WinError 5] Access is denied: 'myDir'
My guess: when I call os.mkdir, sometimes shutil.rmtree hasn't finished execution/ hasn't released the permission for the directory. Hence, the error.
Is there any way to ensure complete execution of shutil.rmtree before calling os.mkdir?
So I encountered the same issue. What I have been using is a pause after shutil.rmtree. I think that pretty much anything that causes your computer to use a clock cycle would do.
All code:
import os
import shutil
import time
dataDir = 'C:/Data/'
if os.path.exists(TEMPDIR):
shutil.rmtree(TEMPDIR)
time.sleep(.0000000000000001)
os.makedirs(TEMPDIR)
If at first you don't succeed...
if os.path.exists(report_path):
shutil.rmtree(report_path)
while True:
try:
os.mkdir(report_path)
break
except PermissionError:
print('Damned Win 10 PERMISSION exception, trying again')
continue
...and if that doesn't work there is at least <ctrl> -c
I have been running into similar issues / the same error message on Windows 10 with calls to shutil.rmtree(dir) being denied access and subsequently not being executed. This in a primitive backup program I wrote which is designed to delete backups over a certain age. I am still testing the below but thus far it works as a temporary workaround in my setups however your mileage may vary.
Running the script as administrator didn't help either. Using os.chmod(f, S_IWRITE) [and/or stat.S_IRUSR | stat.S_IRGRP | stat.S_IROTH | stat.S_IXUSR | stat.S_IRUSR | stat.S_IWUSR | stat.S_IWGRP | stat.S_IXGRP] also got the same refusal by the OS. Changing Controlled Folder Access (which I suspect may be causing this some of the time in Windows 10) and also modifying read only properties on the respective directories also did not help. Neither did changing ownership nor write access in the security tab help.
The workaround I use at the moment is to install, import and use the 'send2trash' module which can be used to remove the directory - assuming your program is otherwise sound - without said access error.
Thereafter one can delete the file from the recycle bin by hand or by automated means(after import of shell from win32com.shell) - obviously this second step will remove all files from the recycle bin. So absolutely check and think BEFORE you run this otherwise you may lose data you don't want to....).
I hope this is helpful.
I have a string of Java source code in Python that I want to compile, execute, and collect the output (stdout and stderr). Unfortunately, as far as I can tell, javac and java require real files, so I have to create a temporary directory.
What is the best way to do this? The tempfile module seems to be oriented towards creating files and directories that are only visible to the Python process. But in this case, I need Java to be able to see them too. However, I also want the other stuff to be handled intelligently if possible (such as deleting the folder when done or using the appropriate system temp folder)
tempfile.NamedTemporaryFile and tempfile.TemporaryDirectory work perfectly fine for your purposes. The resulting objects have a .name attribute that provides a file system visible name that java/javac can handle just fine, just make sure to:
Set the suffix appropriately if the compiler insists on files being named with a .java extension
Always call .flush() on the file handle before handing the .name of a NamedTemporaryFile to an external process or it may (usually will) see an incomplete file
If you don't want Python cleaning up the files when you close the objects, either pass delete=False to NamedTemporaryFile's constructor, or use the mkstemp and mkdtemp functions (which create the objects, but don't clean them up for you).
So for example, you might do:
# Create temporary directory for source and class files
with tempfile.TemporaryDirectory() as d:
# Write source code
srcpath = os.path.join(d.name, "myclass.java")
with open(srcpath, "w") as srcfile:
srcfile.write('source code goes here')
# Compile source code
subprocess.check_call(['javac', srcpath])
# Run source code
# Been a while since I've java-ed; you don't include .java or .class
# when running, right?
invokename = os.path.splitext(srcpath)[0]
subprocess.check_call(['java', invokename])
... with block for TemporaryDirectory done, temp directory cleaned up ...
tempfile.mkstemp creates a file that is normally visible in the filesystem and returns you the path as well. You should be able to use this to create your input and output files - assuming javac will atomically overwrite the output file if it exists there should be no race condition if other processes on your system don't misbehave.
I' trying to create a directory and then delete it (for testing purposes, which I will ommit, but can give details if needed).
Like this:
>>> import os
>>> os.makedirs('C:\\ProgramData\\dir\\test')
>>> os.remove('C:\\ProgramData\\dir\\test')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
PermissionError: [WinError 5] Access is denied: 'C:\\ProgramData\\dir\\test'
I always get access denied, although I'm running the interpreter as an admin. Also I have no problem manually deleting the directory.
Use os.rmdir to delete a folder.
os.remove is for deleting files.
Use os.rmdir to remove a directory. On Windows this is implemented by calling the WinAPI function RemoveDirectory. os.remove is implemented by calling DeleteFile, which is only meant for deleting files. If the filename argument is a directory, the call fails and it sets the last error code to ERROR_ACCESS_DENIED, for which Python 3 raises a PermissionError.
In this case the access denied error is based on the NTSTATUS code STATUS_FILE_IS_A_DIRECTORY, i.e. RtlNtStatusToDosError(0xC00000BA) == 5. Often the kernel's status code is more informative than the corresponding WinAPI error, but not always, and there isn't always a simple mapping from one to the other, depending on the division of labor between the kernel, system processes, services, environment subsystems, and the application. In this case I think the kernel status code is undeniably more informative than a generic access denied error.
At a lower level the cause of the error when trying to delete a directory via DeleteFile is that it calls the system service NtOpenFile with the FILE_NON_DIRECTORY_FILE flag set in OpenOptions, whereas RemoveDirectory specifies FILE_DIRECTORY_FILE. Subsequently both functions call NtSetInformationFile to set the FileDispositionInformation to delete the file or directory.
Just to be a contrarian, let's implement the entire sequence using only file operations on an NTFS file system.
>>> import os, pathlib
>>> base = pathlib.Path(os.environ['ProgramData'])
Create the 'dir' directory:
>>> dirp = base / 'dir::$INDEX_ALLOCATION'
>>> open(str(dirp), 'w').close()
>>> os.path.isdir(str(dirp))
True
By manually specifying the stream type as $INDEX_ALLOCATION, opening this 'file' actually creates an NTFS directory. Incidentally, you can also add multiple named $DATA streams to a directory. Refer to the file streams topic.
Next create the 'test' subdirectory and call os.remove to delete it:
>>> test = base / 'dir' / 'test::$INDEX_ALLOCATION'
>>> open(str(test), 'w').close()
>>> os.path.isdir(str(test))
True
>>> os.remove(str(test))
>>> os.path.exists(str(test))
False
You may be surprised that this worked. Remember the filename in this case explicitly specifies the $INDEX_ALLOCATION stream. This overrules the FILE_NON_DIRECTORY_FILE flag. You get what you ask for. But don't rely on this since these streams are an implementation detail of NTFS, which isn't the only file system in use on Windows.
I have a program where I am creating a multitude of LaTeX files one by one. It is important when creating these LaTeX files to check that they can actually compile to a .pdf without error.
To do so it uses subprocess.call(['pdflatex', '-halt-on-error', tex_file_name]).
Which returns 0 on a successful compile from a .tex to a .pdf, and a 1 otherwise.
The problem I am having is that the only circumstance under which this line of code does not do what I think it should do, is when py.test runs it. If I run this code from an interpreter, or running a script from the command line, it works. But py.test doesn't.
When py.test errors, it leaves behind a log file created by pdflatex, which has this error in it:
{c:/texlive/2012/texmf-var/fonts/map/pdftex/updmap/pdftex.map
!pdfTeX error: pdflatex.exe (file c:/texlive/2012/texmf-var/fonts/map/pdftex/up
dmap/pdftex.map): fflush() failed (Bad file descriptor)
==> Fatal error occurred, no output PDF file produced!
I am hazarding a guess here that py.test is doing some thing with the .tex file prior to pdflatex being able to compile it. But I don't know what.
Temporary files and directories are talked about in the py.test docs. I don't know if they are relevant to my problem, but I have only played around with them briefly.
In case you want to look at the code, a test case looks like this:
from a import Foo
from b import Tree
from latex_tester import latex_tester
def test_Foo():
q1 = foo.Foo()
latex_tester(Tree(1, q1))
and latex_tester looks like this:
import uuid
import os
import subprocess
def latex_tester(tree):
""" Test whether latex is compilable to a PDF.
"""
full_path = r'some_path'
uid = str(uuid.uuid1())
file_name = os.path.join(full_path, 'test' + uid + '.tex')
with open(file_name, 'w') as f:
_write_tree(f, tree)
retcode = subprocess.call(['pdflatex', '-halt-on-error', file_name])
if retcode != 0:
raise RuntimeError("This latex could not be compiled.")
Oddly enough, using 'xelatex' instead of 'pdflatex' makes things work as normal.
For any future readers - I have TeXworks installed which presumably installed both these tools. I don't know if xelatex influences the final pdf produced. It seems to be producing a good .pdf
Anyway, I made this answer to my own question since there doesn't seem to be anything else coming and it certainly solved my problem.
I had exactly the same problem.
I'm using C# as programming language for creating the .tex documents and it crashed during pdflatex when I included a image into the pdf.
And it worked if i started it manually...
Error:
pdfTeX warning: pdflatex
!pdfTeX error: pdflatex (file <linktoFile>/file.pdf): fflush() failed (Bad file descriptor)
Unfortunately xelatex didn't work either so I searched and eventually stumbled upon it.
Basically the error happened for me at this line:
string tex = tex.Replace("\uFFFD\uFFFDMEMNAME\uFFFD\uFFFD", user.Surname)
Here user.Surname was null.
When the tex string is saved into the file it mysteriously remembers the null and pdflatex crashes somewhere completely different. If you on the other hand start pdflatex on the same file again manually the null is gone and it works.
The whole mess went away when I entered a Surname and it works now from the program.
Maybe this will help someone with the same problem.