I am playing with cgi (uploading file form),
and I am receiving the files as storage object and I sotred it in (input) variable.
this is the simple iteration.
for file in input:
filepath = ....
filename, fileext = os.path.splitext(filepath)
file_real_name = ....
file_size = ....
file_type = ...
file_url = ....
file_short_name = ...
file_show_link = ....
# etc
it would be easy if it was only one file , but what If i have more than one ?
how can I have another value that holds all the iteration information in
like uploaded_files where I can access each uploaded file with all the information for the above iteration ?
I tried to read the docs but I cant wrap my head around some iteration concepts yet, sorry :)
You want to use a data structure to hold your data. Depending on the complexity, you may want to simply use a list of dictionaries:
files = []
for file in input:
files.append({
"path": get_path(file),
"name": get_name(file),
"size": get_size(file),
...
})
Or, if you find you need to perform lots of operations on your data, you might want to make your own class and make a list of objects:
class SomeFile:
def __init__(self, path, name, size, ...):
self.path = path
...
def do_something_with_file(self):
...
files = []
for file in input:
files.append(SomeFile(get_path(file), get_name(file), get_size(file), ...))
Note that here you are following a pattern of building up a list by iterating over an iterator. You can do this efficiently using a list comprehension, e.g:
[{"path": get_path(file), "name": get_name(file), ...} for file in input]
Also note that file and input are really bad variable names, as they will mask the builtins file() and input().
results = []
for i in range(5):
file_data = {}
file_data['a'] = i
file_data['b'] = i**2
results.append(file_data)
print results
Related
I'm trying to process a json file like below and extract its data in the below output format for further processing.
json file
{
"application_robotics-2.1.610.80350109": [
"/home/machine_process/application_robotics/services/linear_service/4.106.50109987/robotics.yaml",
"/home/machine_process/application_robotics/services/linear_service/4.106.50109987/application_robotics-4.106.50109987.zip"
],
"web_robotics-3.116.50100987": [
"/home/machine_process/application_robotics/services/web_robotics/3.116.50100987/robotics.yaml",
"/home/machine_process/application_robotics/services/web_robotics/3.116.50100987/web_robotics-3.116.50100987.zip"
]
}
Expected output format
name = "application_robotics-2.1.610.80350109" # where name is a variable to be used in the other portion of the code.
yaml = "/home/machine_process/application_robotics/services/linear_service/4.106.50109987/robotics.yaml" # where yaml is a variable.
zip = "/home/machine_process/application_robotics/services/linear_service/4.106.50109987/application_robotics-4.106.50109987.zip" # where zip is a variable.
same format applied for other entries.
Below is the snippet code I've come up with and I'm not exactly getting the logic. Any help will be really helpful here. Thanks.
with concurrent.futures.ProcessPoolExecutor() as executor:
with open(file_path, "r") as input_json:
json_data = json.load(input_json)
for key, value in json_data.items():
name = json_data[key]
yaml = json_data[value]
zip = json_data[value]
file_location = os.path.dirname(tar)
futures = executor.submit(
other_function_name, yaml, zip, file_location, name
)
results.append(futures)
Current Output:
['home/machine_process/application_robotics/services/linear_service/4.106.50109987/robotics.yaml', '/home/machine_process/application_robotics/services/linear_service/4.106.50109987/application_robotics-4.106.50109987.zip']
Since name corresponds to the keys; yaml to the first element of lists; and zip_ to the second elements (note that zip is a python builtin, so avoid using it as a variable name), we can directly unpack it as we loop over the dictionary and pass these to executor.
with concurrent.futures.ProcessPoolExecutor() as executor:
with open(file_path, "r") as input_json:
json_data = json.load(input_json)
for name, (yaml, zip_) in json_data.items():
file_location = os.path.dirname(tar)
futures = executor.submit(other_function_name, yaml, zip_, file_location, name)
results.append(futures)
i'm trying to append text to file in specific location.
i want to create program which takes input from user(name, image, id) and adds them to this file
names = []
images = []
id = 0
url = ['https://somewebsiteUsingId10.com',
'https://somewebsiteUsingId20.com']
if id == 5:
names.append("Testing Names")
images.append("Testing Images")
elif id == 0:
names.append("Testing one names")
images.append("Testing one Images")
I want modified file to be like this:
names = []
images = []
id = 0
url = ['https://somewebsiteUsingId20.com',
'https://somewebsiteUsingId10.com',
'https://somewebsiteUsingId50.com']
if id == 5:
names.append("Testing Names")
images.append("Testing Images")
elif id == 0:
names.append("Testing one names")
images.append("Testing one Images")
elif id == 50:
names.append("User input")
images.append("User Input")
Thanks!
In cases like this, a good course of action is to put the variable data in a configuration file.
On start-up, your program reads the configuration file and processes it.
Another program can update the configuration file.
Python has the json module in its standard library. This supports lists and dicts, so it is a good match for Python data structures.
Say you write a file urls.json, looking like this:
[
"https://somewebsiteUsingId20.com",
"https://somewebsiteUsingId10.com",
"https://somewebsiteUsingId50.com"
]
In your program you can then do:
import json
with open("urls.json") as f:
urls = json.load(f)
The variable urls now points to a list containing the aforementioned URLs.
Writing the config data goes about the same:
urls = [
"https://www.parrot.org",
"https://www.ministryofsillywalks.org",
"https://www.cheese.net",
]
with open("newurls.json", "w") as f:
json.dump(urls, f, indent=4)
The file newurls.json now contains:
[
"https://www.parrot.org",
"https://www.ministryofsillywalks.org",
"https://www.cheese.net"
]
Note that JSON is pretty flexible, you are not limited to strings:
import datetime
config = {
'directories': ["https://www.parrot.org", "https://www.ministryofsillywalks.org"],
'saved': str(datetime.datetime.now()),
'count': 12
}
with open("configuration.json", "w") as cf:
json.dump(config, cf, indent=4)
This would result in something like:
{
"directories": [
"https://www.parrot.org",
"https://www.ministryofsillywalks.org"
],
"saved": "2022-02-07 21:21:14.787420",
"count": 12
}
(You'd get another date/time, of course.)
The only major downside to JSON files is that they don't allow comments. If you need comments, use another format like the configparser module.
Note that there are other methods like shelve and read&eval but those have potential safety issues.
I'm writing a to-do list application, and to store the class objects task I'm pickling a list of the objects created. However, when I load the data, the list appears empty. The way I structured it is to create an empty list each session, then append the contents of the pickle file. When new tasks are created, they are appended and the whole list is then appended and then reloaded.
This is my first real software project, so my code looks pretty rough. I reviewed it and can't find any glaring errors, but obviously I am doing something wrong.
Here is the relevant code:
import _pickle as pickle
import os.path
from os import path
from datetime import datetime
#checks if data exists, and creates file if it does not
if path.exists('./tasks.txt') != True:
open("./tasks.txt", 'wb')
else:
pass
#define class for tasks
class task:
def __init__(self, name, due, category):
self.name = name
self.due = datetime.strptime(due, '%B %d %Y %I:%M%p')
self.category = category
def expand(self): # returns the contents of the task
return str(self.name) + " is due in " + str((self.due - datetime.now()))
data = []
# load data to list
def load_data():
with open('tasks.txt', 'rb') as file:
while True:
data = []
try:
data.append(pickle.load(file))
except EOFError:
break
...
# returns current task list
def list_tasks():
clear()
if not data:
print("Nothing to see here.")
else:
i = 1
for task in data:
print("%s. %s" % (i, task.expand()))
i = i+1
#define function to add tasks
def addTask(name, due, category):
newTask = task(name, due, category)
data.append(newTask)
with open('./tasks.txt', 'wb') as file:
pickle.dump(data, file)
load_data()
list_tasks()
...
load_data()
list_tasks()
startup()
ask()
data = []
# load data to list
def load_data():
with open('tasks.txt', 'rb') as file:
while True:
data = []
try:
data.append(pickle.load(file))
except EOFError:
break
That second data = [] doesn't look right. Having data = [] both inside and outside of the function creates two data objects, and the one you're appending to won't be accessible anywhere else. And even if it was accessible, it would still be empty since it's being reset to [] in every iteration of the while loop. Try erasing the inner data = []. Then the data.append call will affect the globally visible data, and its contents won't be reset in each loop.
Additionally, going by the rest of your code it looks like that data is supposed to be a list of tasks. But if you pickle a list of tasks and then run data.append(pickle.load(file)), then data will be a list of lists of tasks instead. One way to keep things flat is to use extend instead of append.
data = []
# load data to list
def load_data():
with open('tasks.txt', 'rb') as file:
while True:
try:
data.extend(pickle.load(file))
except EOFError:
break
I think it may also be possible to load the data with a single load call, rather than many calls in a loop. It depends on whether your tasks.txt file is the result of a single pickle.dump call, or if you appended text to it multiple times with multiple pickle.dump calls while the file was opened in "append" mode.
def load_data():
with open('tasks.txt', 'rb') as file:
return pickle.load(file)
data = load_data()
I have a large for loop in which I create json objects and I would like to be able to stream write the object in each iteration to a file. I would like to be able to use the file later in a similar fashion later (read objects one at a time).
My json objects contain newlines and I can't just dump each object as a line in a file.
How can I achieve this?
To make it more concrete, consider the following:
for _id in collection:
dict_obj = build_dict(_id) # build a dictionary object
with open('file.json', 'a') as f:
stream_dump(dict_obj, f)
stream_dump is the function that I want.
Note that I don't want to create a large list and dump the whole list using something like json.dump(obj, file). I want to be able to append the object to the file in each iteration.
Thanks.
You need to work with a subclass of JSONEncoder and then proxy the build_dict function
from __future__ import (absolute_import, division, print_function,)
# unicode_literals)
import collections
import json
mycollection = [1, 2, 3, 4]
def build_dict(_id):
d = dict()
d['my_' + str(_id)] = _id
return d
class SeqProxy(collections.Sequence):
def __init__(self, func, coll, *args, **kwargs):
super(SeqProxy, *args, **kwargs)
self.func = func
self.coll = coll
def __len__(self):
return len(self.coll)
def __getitem__(self, key):
return self.func(self.coll[key])
class JsonEncoderProxy(json.JSONEncoder):
def default(self, o):
try:
iterable = iter(o)
except TypeError:
pass
else:
return list(iterable)
# Let the base class default method raise the TypeError
return json.JSONEncoder.default(self, o)
jsonencoder = JsonEncoderProxy()
collproxy = SeqProxy(build_dict, mycollection)
for chunk in jsonencoder.iterencode(collproxy):
print(chunk)
Ouput:
[
{
"my_1"
:
1
}
,
{
"my_2"
:
2
}
,
{
"my_3"
:
3
}
,
{
"my_4"
:
4
}
]
To read it back chunk by chunk you need to use JSONDecoder and pass a callable as object_hook. This hook will be called with each new decoded object (each dict in your list) when you call JSONDecoder.decode(json_string)
Since you are generating the files yourself, you can simply write out one JSON object per line:
for _id in collection:
dict_obj = build_dict(_id) # build a dictionary object
with open('file.json', 'a') as f:
f.write(json.dumps(dict_obj))
f.write('\n')
And then read them in by iterating over lines:
with open('file.json', 'r') as f:
for line in f:
dict_obj = json.loads(line)
This isn't a great general solution, but it's a simple one if you are both the generator and consumer.
Simplest solution:
Remove all whitespace characters from your json document:
import string
def remove_whitespaces(txt):
""" We shall remove all whitespaces"""
for chr in string.whitespace:
txt = txt.replace(chr)
Obviously you could also json.dumps(json.loads(json_txt)) (BTW this also verify that the text is a valid json).
Now you could write you documents to a file one line each.
Second solution:
Create an [AnyStr]Io stream, write in the Io a valid document, (your documents being part of an object or list) and then write the io in a file (or upload it to the cloud).
First I'd like to mention that I am completely new to Python and I've found it a bit difficult to transition from C++. I apologize if my question comes off as elementary.
I have a class for 'songs' which I have initialized as following. It takes in data from a file that contains a song's ID, name, genre etc. all separated by ::.
def __init__(self):
self.song_names = dict()
self.song_genres = dict()
def load_songs(self, song_id):
f = open(song_id)
for line in f:
line = line.rstrip()
component = line.split("::")
sid = components[0]
same= components[1]
sgenre=components[2]
self.song_names[mid] = sname
self.song_genres[mid] = sgenre
f.close()
The program also takes in data from a file with 'users' information, separated as
UserID::Gender::Age::Occupation::Zip etc. and a file with 'ratings'.
I would I implement a function like def set_song(sid, list((title,genres)))
and something like delete_song(sid) ?
I'm going to have to wind up doing a ton more other functions, but if someone could help me with those two - at least to have a better idea of structure and syntax - handling the others should be easier.
Why not just inherit from dict and use its interface? That way you can use Python's standard mapping operations instead of rolling your own:
class Songs(dict):
def load(self, song_id):
with open(song_id, 'r') as f:
for line in f:
sid, name, genre = line.rstrip().split('::')[:3]
self[sid] = [name, genre]
mysongs = Songs()
mysongs.load('barnes_and_barnes__fish_heads')
mysongs['barnes_and_barnes__fish_heads'] = ['roly poly', 'strange'] # set
del mysongs['barnes_and_barnes__fish_heads'] # delete