Here is my vectorized version:
import pandas as pd
import numpy as np
alpha=.001
data = [ (2, 3), (4, 7), (6, 11), (8, 17), (10, 23), (12, 31), (14, 39), (16, 49), (18, 59), (20, 71), (22, 83), (24, 97), (26, 113), (28, 131), (30, 149), (32, 169), (34, 191), (36, 214), (38, 239), (40, 266), (42, 295), (44, 326), (46, 359), (48, 394), (50, 431)]
##CREATE EXAMPLES MATRIX
x_coordinates = [x[0] for x in data]
x_coords=[]
[x_coords.append([1,x]) for x in x_coordinates]
#Creates a list of all x-coordinates with a 1 column
examples=pd.DataFrame(x_coords).transpose()
#uses that list to create a dataframe. Must transpose so it is dimsion 2,25. rows are features, columns are specific examples.
##CREATE THETA MATRIX/VECTOR
theta_list = [1, 2]
theta = pd.DataFrame(theta_list).transpose()
#creates a df of dimension 1,2.
##CREATE Y VECTOR/MATRIX
y_coordinates = [x[1] for x in data]
y=pd.DataFrame(y_coordinates).transpose()
deriv=pd.DataFrame([])
count=0
while (deriv != 0).all().all() and count <= 500000:
length=len(data)
#theta*X
thetaX=theta.dot(examples)
error=thetaX-y
error_pt2=error.dot(examples.T)
deriv=alpha*(1/length)*error_pt2
theta=theta-deriv
print(theta)
count+=1
print(count)
Here is my version made with loops:
total=0
th0=0
th1=0
alpha=0.001
deriv0=1
deriv1=1
count=0
while deriv0 and deriv1 != 0 and count<=1000000:
total0=0
total1=0
#th0
for i in data:
hyp=th0+(th1*i[0])
#print("Hyp is {}".format(hyp))
total0+=(hyp-i[1])
deriv0=(1/25)*total0
th0temp=th0-(alpha*(deriv0))
#th1
for i in data:
hyp=th0+(th1*i[0])
total1+=(hyp-i[1])*i[0]
deriv1=(1/25)*total1
th1temp=th1-(alpha*(deriv1))
th0=th0temp
th1=th1temp
th0temp=0
th1temp=0
count+=1
print("Theta 0: {} \n Theta 1: {} \n\n".format(th0,th1))
print(count)
When I run the vectorized version, it takes almost 10 times as long to run. I would have expected using vectors would make this code much more efficient than running multiple loops. What gives? Is it just the computational overhead of Pandas which is making this run slower? Maybe Pandas isn't suited for this sort of algorithm
I have a list of tuples with character spans. But there are instances where there is an overlap of the spans.
My aim is to modify the tuple list in such a way that for overlaps only the larger span values is kept and smaller span deleted
Example:
Original list: [(2, 3), (7, 9), (10, 11), (10, 12), (15, 17), (16, 17), (20, 21), (20, 29), (21, 28)]
Modified list: [(2, 3), (7, 9), (10, 12), (15, 17), (20, 29)]
Here (10,11) , (16,17) , (20,21) , (21,28) were removed as they had a bigger span overlap with (10,12) , (15,17) and (20,29) respectively.
I found some answers which deal with the overlap like this but these don't deal with the larger span thing.
My thought was to sort on length of the span difference in descending order and then search for overlaps somehow. This search for overlap is something I cannot figure out
Code
values = [(2, 3), (7, 9), (10, 11), (10, 12), (15, 17), (16, 17), (20, 21), (20, 29), (21, 28)]
outputs = []
for value in values:
flag = True
outputsLoop = outputs[:]
for output in outputsLoop:
fromVal, toVal = output
if value[0] in range(fromVal,toVal+1) or value[1] in range(fromVal,toVal+1):
if (value[1]-value[0]) > (toVal - fromVal):
outputs.remove(output)
else:
flag = False
if flag == True:
outputs.append(value)
print(outputs)
Output
[(2, 3), (7, 9), (10, 12), (15, 17), (20, 29)]
Code explanation
The code loops through the values, and checks them for each part in the checked numbers, and keeps the value with the bigger span. It is not the most efficient or the cleanist code if I had to guess, but it works. And I do believe I can probably optimize it further when I have time.
I try to find a k closest points to a target. For example, I have this set of point:
points = [(0,0), (19, 8), (23, 11), (25, 22), (30, 26), (23, 20), (30,0)]
And I want to find the k-closest point for each element of the list. I have tried this code:
def kClosest(target, points,k):
return sorted(points, key=lambda x:(target[0]-x[0])**2+(target[1]-x[1])**2)[:k]
for i in points:
points_copy = points.copy()
points_copy.remove(i)
print(i, ": ", kClosest(i,points_copy,3))
The output will be like this (which I expect):
(0, 0) : [(19, 8), (23, 11), (30, 0)]
(19, 8) : [(23, 11), (23, 20), (30, 0)]
(23, 11) : [(19, 8), (23, 20), (25, 22)]
(25, 22) : [(23, 20), (30, 26), (23, 11)]
(30, 26) : [(25, 22), (23, 20), (23, 11)]
(23, 20) : [(25, 22), (23, 11), (30, 26)]
(30, 0) : [(23, 11), (19, 8), (23, 20)]
Now I want to modify the output to result in the index of the points. So, I hope the output for point (0,0) is [1, 2, 6]. How can we modify the code to produce the expected output?
Take a look at the Neighbors module from sklearn:
https://scikit-learn.org/stable/modules/classes.html#module-sklearn.neighbors
This will provide you optimized methods to do this.
I also think that you should use some optimized module like the one from Sklearn as stated in #aurelien_morel answer.
In any case a very simple solution for what you are asking is to work with indexes of the list points and use them to get and order by the values and return them.
points = [(0,0), (19, 8), (23, 11), (25, 22), (30, 26), (23, 20), (30,0)]
def kClosest(target, points,k):
return sorted(range(len(points)), key=lambda idx:(target[0]-points[idx][0])**2+(target[1]-points[idx][1])**2)[:k]
for i in points:
points_copy = points.copy()
points_copy.remove(i)
print(i, ": ", kClosest(i,points_copy,3))
This is a rather primitive method of getting what you need, but doesn’t involve using sklearn: before calling kClosest() add the index of the point to its value, like this:
points_with_index = [l+[i] for i,l in enumerate(points)]
Use points_with_index in your call to kClosest() instead of points. Now when you print value from points_with_index use value[2] to show just the original index.
I've got a simple graph created using networkx.
import networkx as nx
import matplotlib.pyplot as plt
from pprint import pprint
G = nx.Graph()
head_nodes = range(0, 9)
tail_nodes = range(1, 10)
edge_ls = list(zip(head_nodes, tail_nodes))
G.add_nodes_from(range(0, 10))
G.add_edges_from(edge_ls)
pprint(G.nodes())
nx.draw(G)
plt.show()
I want to remove the edge between node 0 and 1 and add three new nodes (say node 10,11,12). Then, edges have
to be created between node 0 and 10, 10 and 11, 11 and 2.
I'm using G.remove_edge(0,1) to remove the edge between node 0 and 1.
Could someone suggest which function can be used to add n new nodes?
Also, if n new nodes are added, will these nodes be numbered automatically?
I intend to do this in a loop, delete an edge that already exists between two nodes and add n new nodes and edges connecting these nodes.
EDIT:
I tried the following to add n new edges
G = nx.Graph()
head_nodes = range(0, 9)
tail_nodes = range(1, 10)
edge_ls = list(zip(head_nodes, tail_nodes))
G.add_nodes_from(range(0, 10))
G.add_edges_from(edge_ls)
head = 0
tail = 1
G.remove_edge(head, tail)
Nnodes = G.number_of_nodes()
newnodes = [head, Nnodes+1, Nnodes+2, Nnodes+3, tail] # head and tail already exists
newedges = [(x, y) for x, y in zip(newnodes[0:len(newnodes)-1], newnodes[1:len(newnodes)])]
G.add_edges_from(newedges)
pprint(G.edges())
Output:
EdgeView([(0, 11), (1, 2), (1, 13), (2, 3), (3, 4), (4, 5), (5, 6), (6, 7), (7, 8), (8, 9), (11, 12), (12, 13)])
Expected Output:
EdgeView([(0, 11), (1, 2), (13, 1), (2, 3), (3, 4), (4, 5), (5, 6), (6, 7), (7, 8), (8, 9), (11, 12), (12, 13)])
I'm not sure why the edge that was added in the order (13,1)(head, tail) is stored as (1,13). Any suggestion on how to preserve the order of head and tail node while adding a new edge?
EDIT2:
replacing nx.Graph() with nx.OrderedGraph() also doesn't help.
A Graph is an undirected graph, where (1, 13) and (13, 1) mean the same thing, the edges have no 'arrows'.
What you want is a DiGraph, meaning a directed graph. See https://networkx.github.io/documentation/stable/reference/classes/index.html
An OrderedGraph is something else - it just means that when you iterate over nodes and edges, they come out in a particular order (similar to lists vs sets).
I am rather new to Python and NetworkX. I need to create a list similar to Edgelist=[(0,1),(0,3),(1,0),(1,2),(1,4),(2,1),(2,5)], which elements represent the starting and ending node of an edge (link) that is in turn part of a network.
Rather than setting them manually, I want Python to create the couples you see in the list by randomly selecting the integer values of (start,end) from an assigned range of values (namely, 0, 999), which represent the node IDs. Then, I want to make sure that every node ID is included at least once in the series of (start,end) values (this means that all my nodes will be connected to at least one other node).
I know I could use random.randint(0, 999) but I don't know how to "nest" it into the creation of a list (perhaps a for loop?). I wish I had some code to show you but this is my first attempt at working with NetworkX!
EDIT
To give you a visual idea of what I mean, here are two images. The first is a regular network (aka lattice), and the second is a random one. The edge list of the first was created manually in order to reproduce a chess table, while the second displays an edge list which is a (manually) shuffled counterpart of the first one. As you see, the nodes are kept in exactly the same locations. Hope this helps a bit more. Thanks!
There is a similar answer but for a complete graph on - How to generate a fully connected subgraph from node list using python's networkx module
In your case, using zubinmehta's answer:
import networkx
import itertools
def complete_graph_from_list(L, create_using=None):
G = networkx.empty_graph(len(L),create_using)
if len(L)>1:
if G.is_directed():
edges = itertools.permutations(L,2)
else:
edges = itertools.combinations(L,2)
G.add_edges_from(edges)
return G
You could build the graph as:
S = complete_graph_from_list(map(lambda x: str(x), range(0,1000)))
print S.edges()
Here is a networkx command that will create a graph such that each node has exactly one edge:
import networkx as nx
G = nx.configuration_model([1]*1000)
If you look into the guts of it, it does the following which answers your question - each node will appear in exactly one edge.
import random
mylist = random.suffle(range(start,end))
edgelist = []
while mylist:
edgelist.append((mylist.pop(),mylist.pop()))
You should guarantee that mylist has even length before going through the popping.
Python has inbuilt library called itertools.
Sample as below as how you achieve what you mentioned:
import itertools
list = [3, 4, 6, 7]
sublist_length = 2
comb = itertools.combinations(list, sublist_length)
This will return comb as an iterator.
You can do comb.next() to get next element in the iterator or iterate over a for loop to get all results as you wanted as below.
for item in comb:
print item
which should output:
(3, 4),
(3, 6),
(3, 7),
(4, 6),
(4, 7),
(6, 7),
I hope this will solve your problem.
For the list creation you can do something like:
import random
max = 999
min = 0
original_values = range(min, max) # could be arbitrary list
n_edges = # some number..
my_edge_list = [(random.choice(original_values), random.choice(original_values))
for _ in range(n_edges)]
To assert you have all values in there you can do the following
vals = set([v for tup in my_edge_list for v in tup])
assert all([v in vals for v in original_values])
The assert will make sure you have the proper representation in your edges. As far as doing your best to make sure you don't hit that assert you can do a couple of things.
Sample without replacement from your list of integers until they are all gone to create a "base network" and then randomly add on more to your hearts desire
Make n_edges sufficiently high that it's very likely your condition will be met. If it's not try again...
Really depends on what you're going to use the network for and what kind of structure you want it to have
EDIT: I have updated my response to be more robust to an arbitrary list of values rather than requiring a sequential list
random.seed(datetime.datetime.now())
from random import randint
# ot generate 100 tuples with randints in range 0-99
li = [(randint(0,99),randint(0,99)) for i in range(100)]
print(li)
[(80, 55), (3, 10), (66, 65), (26, 23), (8, 72), (83, 25), (24, 99), (72, 9), (52, 76), (72, 68), (67, 25), (72, 18), (94, 62), (7, 62), (49, 94), (29, 89), (11, 38), (52, 51), (19, 32), (20, 85), (56, 61), (4, 40), (97, 58), (82, 2), (50, 82), (77, 5), (2, 9), (2, 46), (39, 4), (74, 40), (69, 15), (1, 77), (45, 58), (80, 59), (85, 80), (27, 80), (81, 4), (22, 33), (77, 60), (75, 87), (43, 36), (60, 34), (90, 54), (75, 3), (89, 84), (51, 93), (62, 64), (81, 50), (15, 60), (33, 97), (42, 62), (83, 26), (13, 33), (41, 87), (29, 63), (4, 32), (6, 14), (79, 73), (95, 4), (41, 16), (96, 64), (15, 28), (35, 13), (35, 82), (77, 16), (63, 27), (75, 37), (11, 52), (21, 35), (37, 96), (9, 86), (83, 11), (5, 42), (34, 32), (17, 8), (65, 55), (58, 19), (90, 40), (18, 75), (29, 14), (0, 11), (25, 68), (34, 52), (22, 8), (12, 53), (16, 49), (73, 54), (78, 80), (74, 60), (40, 68), (69, 20), (37, 38), (74, 60), (53, 90), (25, 48), (44, 52), (49, 27), (28, 35), (29, 94), (35, 60)]
Here is a solution that first generates a random population of nodes (pop1), then shuffles it (pop2) and combines those into a list of pairs.
Note: this method only yields vertices where each node is exactly once start and exactly once end, so maybe not what you're after. See below for another method
import random, copy
random.seed() # defaults to time.time() ...
# extract a number of samples - the number of nodes you want
pop1 = random.sample(xrange(1000), 10)
pop2 = copy.deepcopy( pop1 )
random.shuffle( pop2 )
# generate pairs from the same population - this guarantees your constraint
pairs = zip( pop1, pop2 )
print pairs
Output:
[(17, 347), (812, 688), (347, 266), (731, 342), (342, 49), (904, 17), (49, 731), (50, 904), (688, 50), (266, 812)]
Here is another method
This allows for duplicate occurrences of the nodes.
The idea is to draw start and end nodes from the same population:
import random
random.seed()
population = range(10) # any population would do
# choose randomly from the population for both ends
# so you can have duplicates
pairs = [(random.choice(population), random.choice(population) for _ in xrange(100)]
print pairs[:10]
Output:
[(1, 9), (7, 1), (8, 6), (4, 7), (6, 2), (7, 3), (0, 2), (1, 0), (8, 3), (8, 3)]