Develop Reference

Using JSON Element for Relative Path File Load in Python

First software job and l inherited a codebase with heavy JSON usage. I'm trying to use Python to load a text file from accessing a JSON element. My relative path understanding is limited but l'm able to load python modules in the same sub directory (not using JSON though). This data file is in the same folder.
I parse and store data within the JSON element here:
with open(cfg) as fp:
edict = json.load(fp)
if edict["dtype"] == "custom" :
data = edict["dtype"]["custom"]["datapath"]
Relevant section of JSON file:
{
"dtype" : {
"custom" : {
"datapath" : "DataPipeLine/example_data.txt",
"type" : "T",
"X" : "encoder",
"Y" : "encoder"
}
}
}
I get an error from passing the data variable into a function later in the program:
UnboundLocalError: local variable referenced before assignment error is raised when you try to assign a value to a local variable before it has been declared.

There are too many errors in your code
I assumed your project directory like this
.
├── DataPipeLine
│   └── example_data.txt
├── cfg.json
└── main.py
There is the right code example
import json
from pathlib import Path
# Get current directory
CUR_DIR = Path(__file__).parent.absolute()
def load_data(path: Path):
print(path.read_text())
# cfg file path
cfg = CUR_DIR / 'cfg.json'
with cfg.open() as fp:
edict = json.load(fp)
# Check there is custom node or not
if edict["dtype"]["custom"]:
# Generate your datapath
data_path = CUR_DIR / edict["dtype"]["custom"]["datapath"]
load_data(data_path)
If my answer solved your problem please accept it as the right answer.

Python to append download link to list of files in Dropbox API

I'm trying to pull a list of dictionaries from Dropbox API for some images and to be represented as follows;
[{name: 'XYZ', url: 'THIS IS A URL'}]
I've use the following which generates a list of file names and path's just fine.
path = ''
files_list = []
def dropbox_list_files(path):
dbx = dropbox_connect()
try:
files = dbx.files_list_folder(path).entries
for file in files:
if isinstance(file, dropbox.files.FileMetadata):
metadata = {
'name': file.name,
'path_lower': file.path_lower,
}
files_list.append(metadata)
But I'm stuck on now creating the sharable links, aswell as then appending these to each of the relevant files in a list.
In the Dropbox API documentation they have the 'sharing_create_shared_link_with_settings' function which looks like it can do this, but requires a file path, which I am not sure if this is just a single path, or how I could A) pass in an iterable of paths for all the files in sequence, and then B) how would I append these together for the purpose explained above?
Any help is much appreciated.

Once you have the files_list populated, loop through and pass the file path to the sharing_create_shared_link method. To do that, you would add something like this to your script to print a list of links..
for file in files_list:
try:
link = dbx.sharing_create_shared_link(file['path_lower'])
print(link.url)
except Exception as e:
print(e)
Assuming you are using the Dropbox SDK, you will want to ensure you have appropriate scopes set (e.g. sharing.write) otherwise will you encounter a permission error.

Folder picker in HTML with Flask for Uploads

I am running a Flask app where the user uploads a file and must select the root folder path of where to upload the file on a network drive. This path is an IIS available network path and is also a network drive on all user's computers.
I want to dynamically show the available folders in HTML, even if new folders are created after the app starts.
I know this can't be done with pure HTML due to security but wanted to know if there was a way around this with Flask. The goal is to use Python to move the upload file to the choosen folder path.
I have tried:
<form><input type="file" name=dir webkitdirectory directory multiple/></form>
But this only works in Chrome. With the path choosen by the user I can pass this onto Python to copy the upload file to there.

Due to modern browser limitations I decided to use JSTree as a solution. And it is working very well. It features a tree structure browser. The structure is the result of outputting the folders as JSON. You can add a search bar as well so the user can just type in a folder name to search.
Please see JSTree https://www.jstree.com/
How to implement this with Flask
HTML/JS:
<link rel="stylesheet" type="text/css" href="https://cdnjs.cloudflare.com/ajax/libs/jstree/3.2.1/themes/default/style.min.css">
<div>
<input class="search-input form-control" placeholder="Search for folder"></input>
</div>
<script id="jstree1" name="jstree1">
/*Search and JS Folder Tree*/
$(function () {
$(".search-input").keyup(function () {
var searchString = $(this).val();
console.log(searchString);
$('#container').jstree('search', searchString);
});
$('#container').jstree({
'core': {
"themes": {
"name": "default"
, "dots": true
, "icons": true
}
, 'data': {
'url': "static/JSONData.json"
, 'type': 'GET'
, 'dataType': 'JSON'
}
}
, "search": {
"case_insensitive": true
, "show_only_matches": true
}
, "plugins": ["search"]
});
});
{ /* --- THIS IS FOLDER SELECTOR FOR ID "folderout" --- */
$("#container").on("select_node.jstree", function (evt, data) {
var number = data.node.text
document.getElementById("folderout").value = number;
});
In Flask/WTForms call on the id "folderout". This will return the path to WTForms when the user clicks the folder.
folderout = TextField('Folder:', validators=[validators.required()])
To Create the JSON JStree File using Python:
import os
# path : string to relative or absolute path to be queried
# subdirs: tuple or list containing all names of subfolders that need to be
# present in the directory
def all_dirs_with_subdirs(path, subdirs):
# make sure no relative paths are returned, can be omitted
path = os.path.abspath(path)
result = []
for root, dirs, files in os.walk(path):
if all(subdir in dirs for subdir in subdirs):
result.append(root)
return result
def get_directory_listing(path):
output = {}
output["text"] = path.decode('latin1')
output["type"] = "directory"
output["children"] = all_dirs_with_subdirs(path, ('Maps', 'Reports'))
return output
with open('test.json', 'w+') as f:
listing = get_directory_listing(".")
json.dump(listing, f)

Python runs on your server, therefoere it will not be possible to use that to move the files on the client side. If you think about it, let's assume you manage to somehow (magically) send python commands to the clients to move files, do you know if they even have python installed to be able to interpret your commands?
Javascript on the other hand is running on client side and was used to achieve this. However, like you said, due to security reasons modern browswers won't allow that. If they would allow it then any website could potentially see your whole File System.
Here is an article that explains a bit why. Look up the File Upload Control section of it. Hope this makes things a bit clearer.
EDIT: after seeing your comment you could achieve that using os.walk. Beware it could be slow.
for root, dirs, files in os.walk(rootPath): # for example "C:/Users/"
for file in files:
if file == (wantedFile):
print(os.path.join(root,file))
break

Python's docx module giving AsserionError when .exe is created

I've a Python file titled my_python_file.py that makes, among other things, a .doc file using the python-docx module. The .doc is created perfectly and gives no problem. The problem comes when I build a .exe of my script and I try to make the .doc. An AssertionError problem appears.
This is my exe maker code (exe_maker.py):
from distutils.core import setup
import py2exe, sys, os
sys.argv.append('py2exe')
setup(
options = {'py2exe': {'bundle_files': 3, 'compressed': True, 'includes': ['lxml.etree', 'lxml._elementpath', 'gzip', 'docx']}},
windows = [{'script': "my_python_file.py"}],
zipfile = None,
)
It seems that moving the python script to a different location produces the error.
File "docx.pyc", line 1063, in savedocx
AssertionError
This is the savedocx line:
document = newdocument()
[...]
coreprops = coreproperties(title=title, subject=subject, creator=creator, keywords=keywords)
approps = appproperties()
contenttypes2 = contenttypes()
websettings2 = websettings()
wordrelationships2 = wordrelationships(relationships)
path_save = "C:\output"
savedocx(document, coreprops, approps, contenttypes2, websettings2, wordrelationships2, path_save)
The savedox is well writen as it works when it's not an .exe file.
How can I make the docx module work correctly? Do I've to add any other path/variable more when I make the exe?
Thanks in advance

I solved the problem by edditing the api.py file of docx egg folder which is located in the Python folder of the system.
Changing this:
_thisdir = os.path.split(__file__)[0]
_default_docx_path = os.path.join(_thisdir, 'templates', 'default.docx')
To this:
thisdir = os.getcwd()
_default_docx_path = os.path.join(thisdir, 'templates', 'default.docx')
The first one was taking the actual running program and adding it to the path to locate the templates folder.
C:\myfiles\myprogram.exe\templates\default.docx
The solution takes only the path, not the running program.
C:\myfiles\templates\default.docx

Instead of changing some library file, I find it easier and cleaner to tell python-docx explicitly where to look for the template, i.e.:
document = Document('whatever/path/you/choose/to/some.docx')
This effectively solves the py2exe and docx path problem.

How to determine the Dropbox folder location programmatically?

I have a script that is intended to be run by multiple users on multiple computers, and they don't all have their Dropbox folders in their respective home directories. I'd hate to have to hard code paths in the script. I'd much rather figure out the path programatically.
Any suggestions welcome.
EDIT:
I am not using the Dropbox API in the script, the script simply reads files in a specific Dropbox folder shared between the users. The only thing I need is the path to the Dropbox folder, as I of course already know the relative path within the Dropbox file structure.
EDIT:
If it matters, I am using Windows 7.

I found the answer here. Setting s equal to the 2nd line in ~\AppData\Roaming\Dropbox\host.db and then decoding it with base64 gives the path.
def _get_appdata_path():
import ctypes
from ctypes import wintypes, windll
CSIDL_APPDATA = 26
_SHGetFolderPath = windll.shell32.SHGetFolderPathW
_SHGetFolderPath.argtypes = [wintypes.HWND,
ctypes.c_int,
wintypes.HANDLE,
wintypes.DWORD,
wintypes.LPCWSTR]
path_buf = wintypes.create_unicode_buffer(wintypes.MAX_PATH)
result = _SHGetFolderPath(0, CSIDL_APPDATA, 0, 0, path_buf)
return path_buf.value
def dropbox_home():
from platform import system
import base64
import os.path
_system = system()
if _system in ('Windows', 'cli'):
host_db_path = os.path.join(_get_appdata_path(),
'Dropbox',
'host.db')
elif _system in ('Linux', 'Darwin'):
host_db_path = os.path.expanduser('~'
'/.dropbox'
'/host.db')
else:
raise RuntimeError('Unknown system={}'
.format(_system))
if not os.path.exists(host_db_path):
raise RuntimeError("Config path={} doesn't exists"
.format(host_db_path))
with open(host_db_path, 'r') as f:
data = f.read().split()
return base64.b64decode(data[1])

There is an answer to this on Dropbox Help Center - How can I programmatically find the Dropbox folder paths?
Short version:
Use ~/.dropbox/info.json or %APPDATA%\Dropbox\info.json
Long version:
Access the valid %APPDATA% or %LOCALAPPDATA% location this way:
import os
from pathlib import Path
import json
try:
json_path = (Path(os.getenv('LOCALAPPDATA'))/'Dropbox'/'info.json').resolve()
except FileNotFoundError:
json_path = (Path(os.getenv('APPDATA'))/'Dropbox'/'info.json').resolve()
with open(str(json_path)) as f:
j = json.load(f)
personal_dbox_path = Path(j['personal']['path'])
business_dbox_path = Path(j['business']['path'])

You could search the file system using os.walk. The Dropbox folder is probably within the home directory of the user, so to save some time you could limit your search to that. Example:
import os
dropbox_folder = None
for dirname, dirnames, filenames in os.walk(os.path.expanduser('~')):
for subdirname in dirnames:
if(subdirname == 'Dropbox'):
dropbox_folder = os.path.join(dirname, subdirname)
break
if dropbox_folder:
break
# dropbox_folder now contains the full path to the Dropbox folder, or
# None if the folder wasn't found
Alternatively you could prompt the user for the Dropbox folder location, or make it configurable via a config file.

This adaptation based on J.F. Sebastian's suggestion works for me on Ubuntu:
os.path.expanduser('~/Dropbox')
And to actually set the working directory to be there:
os.chdir(os.path.expanduser('~/Dropbox'))

Note: answer is valid for Dropbox v2.8 and higher
Windows
jq -r ".personal.path" < %APPDATA%\Dropbox\info.json
This needs jq - JSON parser utility to be installed. If you are happy user of Chocolatey package manager, just run choco install jq before.
Linux
jq -r ".personal.path" < ~/.dropbox/info.json
Just similarly to Windows install jq using package manager of your distro.

Note: requires Dropbox >= 2.8
Dropbox now stores the paths in json format in a file called info.json. It is located in one of the two following locations:
%APPDATA%\Dropbox\info.json
%LOCALAPPDATA%\Dropbox\info.json
I can access the %APPDATA% environment variable in Python by os.environ['APPDATA'], however I check both that and os.environ['LOCALAPPDATA']. Then I convert the JSON into a dictionary and read the 'path' value under the appropriate Dropbox (business or personal).
Calling get_dropbox_location() from the code below will return the filepath of the business Dropbox, while get_dropbox_location('personal') will return the file path of the personal Dropbox.
import os
import json
def get_dropbox_location(account_type='business'):
"""
Returns a string of the filepath of the Dropbox for this user
:param account_type: str, 'business' or 'personal'
"""
info_path = _get_dropbox_info_path()
info_dict = _get_dictionary_from_path_to_json(info_path)
return _get_dropbox_path_from_dictionary(info_dict, account_type)
def _get_dropbox_info_path():
"""
Returns filepath of Dropbox file info.json
"""
path = _create_dropox_info_path('APPDATA')
if path:
return path
return _create_dropox_info_path('LOCALAPPDATA')
def _create_dropox_info_path(appdata_str):
r"""
Looks up the environment variable given by appdata_str and combines with \Dropbox\info.json
Then checks if the info.json exists at that path, and if so returns the filepath, otherwise
returns False
"""
path = os.path.join(os.environ[appdata_str], r'Dropbox\info.json')
if os.path.exists(path):
return path
return False
def _get_dictionary_from_path_to_json(info_path):
"""
Loads a json file and returns as a dictionary
"""
with open(info_path, 'r') as f:
text = f.read()
return json.loads(text)
def _get_dropbox_path_from_dictionary(info_dict, account_type):
"""
Returns the 'path' value under the account_type dictionary within the main dictionary
"""
return info_dict[account_type]['path']
This is a pure Python solution, unlike the other solution using info.json.

One option is you could go searching for the .dropbox.cache directory which (at least on Mac and Linux) is a hidden folder in the Dropbox directory.
I am fairly certain that Dropbox stores its preferences in an encrypted .dbx container, so extracting it using the same method that Dropbox uses is not trivial.

This should work on Win7. The use of getEnvironmentVariable("APPDATA") instead of os.getenv('APPDATA') supports Unicode filepaths -- see question titled Problems with umlauts in python appdata environvent variable.
import base64
import ctypes
import os
def getEnvironmentVariable(name):
""" read windows native unicode environment variables """
# (could just use os.environ dict in Python 3)
name = unicode(name) # make sure string argument is unicode
n = ctypes.windll.kernel32.GetEnvironmentVariableW(name, None, 0)
if not n:
return None
else:
buf = ctypes.create_unicode_buffer(u'\0'*n)
ctypes.windll.kernel32.GetEnvironmentVariableW(name, buf, n)
return buf.value
def getDropboxRoot():
# find the path for Dropbox's root watch folder from its sqlite host.db database.
# Dropbox stores its databases under the currently logged in user's %APPDATA% path.
# If you have installed multiple instances of dropbox under the same login this only finds the 1st one.
# Dropbox stores its databases under the currently logged in user's %APPDATA% path.
# usually "C:\Documents and Settings\<login_account>\Application Data"
sConfigFile = os.path.join(getEnvironmentVariable("APPDATA"),
'Dropbox', 'host.db')
# return null string if can't find or work database file.
if not os.path.exists(sConfigFile):
return None
# Dropbox Watch Folder Location is base64 encoded as the last line of the host.db file.
with open(sConfigFile) as dbxfile:
for sLine in dbxfile:
pass
# decode last line, path to dropbox watch folder with no trailing slash.
return base64.b64decode(sLine)
if __name__ == '__main__':
print getDropboxRoot()