Related
I have an array of lines, with each line being represented by:
{
'ms': int,
'e_up': bool,
'e_down': bool,
'f_up': bool,
'f_down': bool,
'l_up': bool,
'l_down': bool,
'r_up': bool,
'r_down': bool,
'b': int,
'a': int,
'c': int,
'd': int
}
I want to loop through all lines (an array of lines, as a dictionary) and find all duplicates and their .ms property.
For example, if I have:
(1902, False, False, False, False, False, False, True, False, 128, -37, -127, -20)
(1843, False, False, False, False, True, False, False, False, 0, 13, -13, 0)
(1968, False, False, False, False, True, False, False, False, 0, 13, -13, 0)
(234, False, False, False, False, True, False, False, False, 0, 13, -13, 0)
(0, False, False, False, False, True, False, False, False, 0, 13, -13, 0)
I want the output to be:
[
[
1843,
1968,
234,
0
]
]
I want to find all possible combinations, and time isn't an issue here, if it took the extra time it wouldn't really matter to me. How can I accomplish this with Python? (No external libraries please)
You can take advantage of the fact the tuples can be used as keys in a dictionary. The following code uses the tuple of values other than 'ms' as a key in a dictionary, and the 'ms' values are saved as a list in the dictionary. Any list with 2 values or more indicates duplicates:
itemlist = list()
itemlist.append((1902, False, False, False, False, False, False, True, False, 128, -37, -127, -20))
itemlist.append((1843, False, False, False, False, True, False, False, False, 0, 13, -13, 0))
itemlist.append((1968, False, False, False, False, True, False, False, False, 0, 13, -13, 0))
itemlist.append((234, False, False, False, False, True, False, False, False, 0, 13, -13, 0))
itemlist.append((0, False, False, False, False, True, False, False, False, 0, 13, -13, 0))
itemdict = dict()
# create dictionary with lists of items according to signature
for item in itemlist:
key = item[1:]
if key in itemdict:
itemdict[key].append(item[0])
else:
itemdict[key] = [item[0]]
# iterate over dictionary and find items with more than one occurence
duplicates = []
for value in itemdict.values():
if len(value)>1:
duplicates.extend(value)
print(duplicates)
The way I solved the problem was by searching each index with every other non-checked index of the array and finding the duplicates.
def find_duplicate(lines, line, duplicates, checked):
if (line['ms'] in checked):
return duplicates, checked
duplicate = list()
duplicate.append(line['ms'])
checked.append(line['ms'])
for i in range(len(lines)):
new_line = lines[i]
if new_line['ms'] in checked: continue
if new_line['e_up'] == line['e_up'] and new_line['e_down'] == line['e_down'] and new_line['f_up'] == line['f_up'] and new_line['f_down'] == line['f_down'] and new_line['l_up'] == line['l_up'] and new_line['l_down'] == line['l_down'] and new_line['r_up'] == line['r_up'] and new_line['r_down'] == line['r_down'] and new_line['b'] == line['b'] and new_line['a'] == line['a'] and new_line['c'] == line['c'] and new_line['d'] == line['d']:
duplicate.append(new_line['ms'])
checked.append(new_line['ms'])
duplicates.append(duplicate)
return duplicates, checked
And then I used the above function on every non-checked index of the potential duplicate (lines) array.
duplicates = list()
checked = list()
for i in range(len(lines)):
duplicates, checked = find_duplicate(lines, lines[i], duplicates, checked)
print(duplicates)
Input to the code:
(1902, False, False, False, False, False, False, True, False, 128, -37, -127, -20)
(1843, False, False, False, False, True, False, False, False, 0, 13, -13, 0)
(1932, False, False, False, False, True, False, False, False, 0, 13, -13, 0)
(1847, False, True, False, False, True, False, False, False, 0, 13, -13, 0)
(1869, False, True, False, False, True, False, False, False, 0, 13, -13, 0)
Output:
[[1902], [1843, 1932], [1847, 1869]]
I did an experiment and I did not get the result I was expecting.
For the first part, I am using
trainloader = torch.utils.data.DataLoader(trainset, batch_size=128,
shuffle=False, num_workers=0)
I save trainloader.dataset.targets to the variable a, and trainloader.dataset.data to the variable b before training my model. Then, I train the model using trainloader. After the training is finished, I save trainloader.dataset.targets to the variable c, and trainloader.dataset.data to the variable d. Finally, I check a == c and b == d and they both give True, which was expected because the shuffle parameter of the DataLoader is False.
For the second part, I am using
trainloader = torch.utils.data.DataLoader(trainset, batch_size=128,
shuffle=True, num_workers=0)
I save trainloader.dataset.targets to the variable e, and trainloader.dataset.data to the variable f before training my model. Then, I train the model using trainloader. After the training is finished, I save trainloader.dataset.targets to the variable g, and trainloader.dataset.data to the variable h. I expect e == g and f == h to be both False since shuffle=True, but they give True again. What am I missing from the definition of DataLoader class?
I believe that the data that is stored directly in the trainloader.dataset.data or .target will not be shuffled, the data is only shuffled when the DataLoader is called as a generator or as iterator
You can check it by doing next(iter(trainloader)) a few times without shuffling and with shuffling and they should give different results
import torch
import torchvision
transform = torchvision.transforms.Compose([
torchvision.transforms.ToTensor(),
])
MNIST_dataset = torchvision.datasets.MNIST('~/Desktop/intern/',download = True, train = False,
transform = transform)
dataLoader = torch.utils.data.DataLoader(MNIST_dataset,
batch_size = 128,
shuffle = False,
num_workers = 10)
target = dataLoader.dataset.targets
MNIST_dataset = torchvision.datasets.MNIST('~/Desktop/intern/',download = True, train = False,
transform = transform)
dataLoader_shuffled= torch.utils.data.DataLoader(MNIST_dataset,
batch_size = 128,
shuffle = True,
num_workers = 10)
target_shuffled = dataLoader_shuffled.dataset.targets
print(target == target_shuffled)
_, target = next(iter(dataLoader));
_, target_shuffled = next(iter(dataLoader_shuffled))
print(target == target_shuffled)
This will give :
tensor([True, True, True, ..., True, True, True])
tensor([False, False, False, False, False, False, False, False, False, False,
False, False, False, False, False, False, False, False, False, False,
False, False, False, False, False, False, False, False, False, True,
False, False, False, False, False, False, False, False, False, False,
False, False, False, False, False, False, False, False, False, False,
False, False, False, False, False, False, False, False, False, False,
False, False, False, False, True, False, False, False, False, False,
False, True, False, False, False, False, False, False, False, False,
False, False, False, False, False, False, False, False, False, False,
False, False, False, False, True, True, False, False, False, False,
False, False, False, False, False, False, False, False, False, False,
False, False, False, False, False, True, False, False, True, False,
False, False, False, False, False, False, False, False])
However the data and label stored in data and target is a fixed list and since you are trying to access it directly, they will not be shuffled.
I was facing a similar issue while loading the data using the Dataset class. I stopped loading the data using Dataset class, and instead use the following code which is working fine with me
X = torch.from_numpy(X)
y = torch.from_numpy(y)
train_data = torch.utils.data.TensorDataset(X, y)
train_loader = torch.utils.data.DataLoader(train_data, batch_size=32, shuffle=True)
where X & y are numpy array from csv file.
This is an issue I'm running while convertinf DQN to Double DQN for the cartpole problem. I'm getting close to figuring it out.
tensor([0.1205, 0.1207, 0.1197, 0.1195, 0.1204, 0.1205, 0.1208, 0.1199, 0.1206,
0.1199, 0.1204, 0.1205, 0.1199, 0.1204, 0.1204, 0.1203, 0.1198, 0.1198,
0.1205, 0.1204, 0.1201, 0.1205, 0.1208, 0.1202, 0.1205, 0.1203, 0.1204,
0.1205, 0.1206, 0.1206, 0.1205, 0.1204, 0.1201, 0.1206, 0.1206, 0.1199,
0.1198, 0.1200, 0.1206, 0.1207, 0.1208, 0.1202, 0.1201, 0.1210, 0.1208,
0.1205, 0.1205, 0.1201, 0.1193, 0.1201, 0.1205, 0.1207, 0.1207, 0.1195,
0.1210, 0.1204, 0.1209, 0.1207, 0.1187, 0.1202, 0.1198, 0.1202])
tensor([ True, True, True, True, True, True, True, True, True, True,
True, True, True, True, True, True, False, True, True, True,
True, True, True, True, True, True, True, False, True, True,
True, True, True, True, True, True, True, True, True, True,
True, True, True, True, True, True, True, True, True, True,
True, True, True, True, True, True, True, True, True, True,
True, True, True, True])
As you can see here two tensors.
The first has the q values I want
but,
some values need to be changed to zeros because of it an end state.
The second tensor shows where it will be zeros.
At the index where the Boolean value is false is the equivalent spot for where the upper tensor needs to be zeros.
I am not sure how to do that.
You can use torch.where - torch.where(condition, x, y)
Ex.:
>>> x = tensor([0.2853, 0.5010, 0.9933, 0.5880, 0.3915, 0.0141, 0.7745,
0.0588, 0.4939, 0.0849])
>>> condition = tensor([False, True, True, True, False, False, True,
False, False, False])
>>> # It's equivalent to `torch.where(condition, x, tensor(0.0))`
>>> x.where(condition, tensor(0.0))
tensor([0.0000, 0.5010, 0.9933, 0.5880, 0.0000, 0.0000, 0.7745,
0.0000, 0.0000,0.0000])
If your above tensor is the value tensor and the bottom one is the decision tensor, then
value_tensor[decision_tensor==False] = 0
Moreover, you could also convert them to numpy arrays and perform the same operation and it should work.
I have a problem with Sublime Text 3 when I saving file with Python syntax. The spaces near to the operators are add automatically.
For example:
I type: print(a," ",b+c) and after saving file it will be: print(a, " ", b + c)
I try change settings but I don't know what does spaces after CTRL+S. I want white spaces but I don't want auto adding spaces in places where I don't type spaces.
My settings file:
{
"always_show_minimap_viewport": true,
"auto_complete": false,
"auto_complete_commit_on_tab": true,
"auto_match_enabled": true,
"bold_folder_labels": true,
"caret_style": "solid",
"color_scheme": "Packages/ayu/ayu-dark.tmTheme",
"detect_indentation": true,
"draw_indent_guides": true,
"draw_minimap_border": true,
"enable_telemetry": false,
"ensure_newline_at_eof_on_save": true,
"folder_exclude_patterns":
[
".svn",
".git",
".hg",
"CVS",
"*.DS_Store",
"*.pyc",
"pycache"
],
"font_face": "Liberation Mono",
"font_options":
[
"subpixel_antialias",
"no_round"
],
"font_size": 14,
"highlight_line": false,
"highlight_modified_tabs": false,
"ignored_packages":
[
"Vintage"
],
"indent_guide_options":
[
"draw_active",
"draw_normal"
],
"indent_to_bracket": true,
"line_padding_bottom": 0,
"line_padding_top": 0,
"match_brackets": true,
"match_brackets_angle": false,
"match_brackets_braces": true,
"match_brackets_content": true,
"match_brackets_square": true,
"new_window_settings":
{
"hide_open_files": true,
"show_tabs": true,
"side_bar_visible": true,
"status_bar_visible": true
},
"pep8_max_line_length": 79,
"preview_on_click": false,
"rulers":
[
79
],
"shift_tab_unindent": true,
"show_panel_on_build": false,
"soda_classic_tabs": true,
"soda_folder_icons": false,
"tab_size": 4,
"theme": "ayu-dark.sublime-theme",
"translate_tabs_to_spaces": true,
"trim_trailing_white_space_on_save": true,
"use_simple_full_screen": true,
"vintage_start_in_command_mode": false,
"wide_caret": true,
"word_wrap": true,
"wrap_width": 80,
"material_theme_accent_graphite": true ,
"material_theme_compact_sidebar": true,
"file_exclude_patterns": ["*.pyc", "*.pyo", "*.exe", "*.dll", "*.obj","*.o", "*.a", "*.lib", "*.so", "*.dylib", "*.ncb", "*.sdf", "*.suo", "*.pdb", "*.idb", ".DS_Store", "*.class", "*.psd", "*.sublime-workspace"],
}
Syntax-settings file:
// These settings override both User and Default settings for the Python syntax
{
"draw_white_space": "all",
"auto_indent": true,
"rulers": [79],
"smart_indent": true,
"tab_size": 4,
"trim_automatic_white_space": true,
"use_tab_stops": true,
"word_wrap": true,
"wrap_width": 80
}
I found a solution. It was a Anaconda plugin.
You must set the following flag in the user's settings of this plugin:
"auto_formatting": true,
"autoformat_ignore":
[
"E309",
"E501",
"E221",
"E222",
"E223",
"E225",
"E226",
"E227",
"E228",
"E231"
]
I couldn't really find this information anywhere, I am looking for a list of possible keys that can be used in the profile.set_preference() API.
Here is some context:
from selenium import webdriver
from pyvirtualdisplay import Display
display = Display(visible=0, size=(1024, 768))
display.start()
profile = webdriver.FirefoxProfile()
Now, if I want to, say specify a client SSL, I need to configure that as a preference of FireFox profile. I am trying to find the list of all the preferences so I can play with this.
You can look at profile.DEFAULT_PREFERENCES which is the json at python2.7/site-packages/selenium/webdriver/firefox/webdriver_prefs.json
{u'frozen': {u'app.update.auto': False,
u'app.update.enabled': False,
u'browser.EULA.3.accepted': True,
u'browser.EULA.override': True,
u'browser.displayedE10SNotice': 4,
u'browser.download.manager.showWhenStarting': False,
u'browser.link.open_external': 2,
u'browser.link.open_newwindow': 2,
u'browser.offline': False,
u'browser.reader.detectedFirstArticle': True,
u'browser.safebrowsing.enabled': False,
u'browser.safebrowsing.malware.enabled': False,
u'browser.search.update': False,
u'browser.selfsupport.url': u'',
u'browser.sessionstore.resume_from_crash': False,
u'browser.shell.checkDefaultBrowser': False,
u'browser.tabs.warnOnClose': False,
u'browser.tabs.warnOnOpen': False,
u'datareporting.healthreport.logging.consoleEnabled': False,
u'datareporting.healthreport.service.enabled': False,
u'datareporting.healthreport.service.firstRun': False,
u'datareporting.healthreport.uploadEnabled': False,
u'datareporting.policy.dataSubmissionEnabled': False,
u'datareporting.policy.dataSubmissionPolicyAccepted': False,
u'devtools.errorconsole.enabled': True,
u'dom.disable_open_during_load': False,
u'extensions.autoDisableScopes': 10,
u'extensions.blocklist.enabled': False,
u'extensions.logging.enabled': True,
u'extensions.update.enabled': False,
u'extensions.update.notifyUser': False,
u'javascript.enabled': True,
u'network.http.phishy-userpass-length': 255,
u'network.manage-offline-status': False,
u'offline-apps.allow_by_default': True,
u'prompts.tab_modal.enabled': False,
u'security.csp.enable': False,
u'security.fileuri.origin_policy': 3,
u'security.fileuri.strict_origin_policy': False,
u'security.warn_entering_secure': False,
u'security.warn_entering_secure.show_once': False,
u'security.warn_entering_weak': False,
u'security.warn_entering_weak.show_once': False,
u'security.warn_leaving_secure': False,
u'security.warn_leaving_secure.show_once': False,
u'security.warn_submit_insecure': False,
u'security.warn_viewing_mixed': False,
u'security.warn_viewing_mixed.show_once': False,
u'signon.rememberSignons': False,
u'toolkit.networkmanager.disable': True,
u'toolkit.telemetry.enabled': False,
u'toolkit.telemetry.prompted': 2,
u'toolkit.telemetry.rejected': True},
u'mutable': {u'browser.dom.window.dump.enabled': True,
u'browser.newtab.url': u'about:blank',
u'browser.newtabpage.enabled': False,
u'browser.startup.homepage': u'about:blank',
u'browser.startup.page': 0,
u'dom.max_chrome_script_run_time': 30,
u'dom.max_script_run_time': 30,
u'dom.report_all_js_exceptions': True,
u'javascript.options.showInConsole': True,
u'network.http.max-connections-per-server': 10,
u'startup.homepage_welcome_url': u'about:blank',
u'webdriver_accept_untrusted_certs': True,
u'webdriver_assume_untrusted_issuer': True}}
I usually just open about:config and search the list of preferences there. There is also this helpful resource which documents part of the preferences.