I have a task to create a program that detects the number plates that are both foreign and speeding. For this, a section of the road, elapsed times etc had to be created. The problem of the code isn't with how unrealistic the speeds are, but the fact that I found out from my teacher recently that the speeds are supposed to be in miles per hour, not metres per second.
import re
# DATA
distance = 0.06 # Distance between the Camera A and B; 0.06 = 600 metres
speed_limit = 20 # (meters per second)
number_plates = ["DV61 GGB", #UK
"D31 EG 2A", #F
"5314 10A02", #F
"24TEG 5063", #F
"TR09 TRE", #UK
"524 WAL 75", #F
"TR44 VCZ", #UK
"FR52 SWD", #UK
"100 GBS 12", #F
"HG55 BPO" #UK
]
enter = [7.12,7.17,7.22,7.12,7.23,7.41,7.18,7.25,7.11,7.38]
leave = [7.56,7.39,7.49,7.56,7.45,7.57,7.22,7.31,7.59,7.47]
# Find the non-UK plates
pattern = "(?![A-Z]{2}\d{2}\s+[A-Z]{3}$)"
foreign_numbers = list(filter(lambda x: re.match(pattern, x), number_plates))
# Calculations for speed
elapsed = [(l - e)/100 for l, e in zip(leave, enter)]
speed = [distance/t for t in converted_elapsed]
# Dictionary for foreign speeders + 2 conditions
foreign_speeders = {plate: speed
for plate, speed in zip(number_plates, speed)
if (plate in foreign_numbers) and (speed > speed_limit)}
print("10 cars have passed Camera A, then Camera B\nSpeed limit is 20 meters per second.\n")
# Write foreign speeders to file
for plate, speed in foreign_speeders.items():
speeders_data = open("speeders.txt","w") # Opens file with name of "speeders.txt"
speeders_data.write(
"{0:>13s} is foreign and is speeding at{1:5.1f} mps, and has an excess speed of {2:3.1f} mps.".format(plate, speed, speed-speed_limit))
speeders_data = open("speeders.txt","r")
print(speeders_data.read())
speeders_data.close()
I wonder, would it be simpler to re-write all speed variables, for example speed_limit and the items of elapsed & formula of speed in their converted forms, or to convert the speed in the middle of the code?
Whichever solution seems more suitable, how can I do it?
As a start, you definitely need a function which can convert units, e.g. mps_to_mph as #mirosval suggested.
However, I suggest you should make it more obvious what the unit is at some point.
Simplest solution is to have it in variable name: speed_mps = 78.7, speed_mph = mps_to_mph(speed_mps), otherwise you will not be able to understand your code when you read it again (well, you might never read this code again, but at least your teacher will... and every code you ever write should be easy to understand without additional explanations).
In a more complex application, with many such calculations, you might want to have a class which can remember units and knows how to convert values, so that you can do something like:
speed = Speed(78.8, 'm/s')
if speed > Speed(60, 'mph'):
# something
You can write a function such as:
def mps_to_mph(mps):
return 2.23694 * mps
and then use it in your for loop:
speed = map(mps_to_mph, speed)
speed_limit = mps_to_mph(speed_limit)
You can leave the calculations and thresholding in meters per second and only convert for the display.
Related
Hey guys I have a script that compares each possible user and checks how similar their text is:
dictionary = {
t.id: (
t.text,
t.set,
t.compare_string
)
for t in dataframe.itertuples()
}
highly_similar = []
for a, b in itertools.combinations(dictionary.items(), 2):
if a[1][2] == b[1][2] and not a[1][1].isdisjoint(b[1][1]):
similarity_score = fuzz.ratio(a[1][0], b[1][0])
if (similarity_score >= 95 and len(a[1][0]) >= 10) or similarity_score == 100:
highly_similar.append([a[0], b[0], a[1][0], b[1][0], similarity_score])
This script takes around 15 minutes to run, the dataframe contains 120k users, so comparing each possible combination takes quite a bit of time, if I just write pass on the for loop it takes 2 minutes to loop through all values.
I tried using filter() and map() for the if statements and fuzzy score but the performance was worse. I tried improving the script as much as I could but I don't know how I can improve this further.
Would really appreciate some help!
It is slightly complicated to reason about the data since you have not attached it, but we can see multiple places that might provide an improvement:
First, let's rewrite the code in a way which is easier to reason about than using the indices:
dictionary = {
t.id: (
t.text,
t.set,
t.compare_string
)
for t in dataframe.itertuples()
}
highly_similar = []
for a, b in itertools.combinations(dictionary.items(), 2):
a_id, (a_text, a_set, a_compre_string) = a
b_id, (b_text, b_set, b_compre_string) = b
if (a_compre_string == b_compre_string
and not a_set.isdisjoint(b_set)):
similarity_score = fuzz.ratio(a_text, b_text)
if (similarity_score >= 95 and len(a_text) >= 10)
or similarity_score == 100):
highly_similar.append(
[a_id, b_id, a_text, b_text, similarity_score])
You seem to only care about pairs having the same compare_string values. Therefore, and assuming this is not something that all pairs share, we can key by whatever that value is to cover much less pairs.
To put some numbers into it, let's say you have 120K inputs, and 1K values for each value of val[1][2] - then instead of covering 120K * 120K = 14 * 10^9 combinations, you would have 120 bins of size 1K (where in each bin we'd need to check all pairs) = 120 * 1K * 1K = 120 * 10^6 which is about 1000 times faster. And it would be even faster if each bin has less than 1K elements.
import collections
# Create a dictionary from compare_string to all items
# with the same compare_string
items_by_compare_string = collections.defaultdict(list)
for item in dictionary.items():
compare_string = item[1][2]
items_by_compare_string[compare_string].append(items)
# Iterate over each group of items that have the same
# compare string
for item_group in items_by_compare_string.values():
# Check pairs only within that group
for a, b in itertools.combinations(item_group, 2):
# No need to compare the compare_strings!
if not a_set.isdisjoint(b_set):
similarity_score = fuzz.ratio(a_text, b_text)
if (similarity_score >= 95 and len(a_text) >= 10)
or similarity_score == 100):
highly_similar.append(
[a_id, b_id, a_text, b_text, similarity_score])
But, what if we want more speed? Let's look at the remaining operations:
We have a check to find if two sets share at least one item
This seems like an obvious candidate for optimization if we have any knowledge about these sets (to allow us to determine which pairs are even relevant to compare)
Without additional knowledge, and just looking at every two pairs and trying to speed this up, I doubt we can do much - this is probably highly optimized using internal details of Python sets, I don't think it's likely to optimize it further
We a fuzz.ratio computation which is some external function, and I'm going to assume is heavy
If you are using this from the FuzzyWuzzy package, make sure to install python-Levenshtein to get the speedups detailed here
We have some comparisons which we are unlikely to be able to speed up
We might be able to cache the length of a_text by nesting the two loops, but that's negligible
We have appends to a list, which runs on average ("amortized") constant time per operation, so we can't really speed that up
Therefore, I don't think we can reasonably suggest any more speedups without additional knowledge. If we know something about the sets that can help optimize which pairs are relevant we might be able to speed things up further, but I think this is about it.
EDIT: As pointed out in other answers, you can obviously run the code in multi-threading. I assumed you were looking for an algorithmic change that would possibly reduce the number of operations significantly, instead of just splitting these over more CPUs.
Essentially, from python programming side, i see two things that can improve your processing time:
Multi-threads and Vectorized operations
From the fuzzy score side, here is a list of tips you can use to improve your processing time (new anonymous tab to avoid paywall):
https://towardsdatascience.com/fuzzy-matching-at-scale-84f2bfd0c536
Using multi thread you can speed you operation up to N times, being N the number of threads in you CPU. You can check it with:
import multiprocessing
multiprocessing.cpu_count()
Using vectorized operations you can parallel process your operations in low level with SIMD (single instruction / multiple data) operations, or with gpu tensor operations (like those in tensorflow/pytorch).
Here is a small comparison of results for each case:
import numpy as np
import time
A = [np.random.rand(512) for i in range(2000)]
B = [np.random.rand(512) for i in range(2000)]
high_similarity = []
def measure(i,j,a,b,high_similarity):
d = ((a-b)**2).sum()
if d>12:
high_similarity.append((i,j,d))
start_single_thread = time.time()
for i in range(len(A)):
for j in range(len(B)):
if i<j:
measure(i,j,A[i],B[j],high_similarity)
finis_single_thread = time.time()
print("single thread time:",finis_single_thread-start_single_thread)
out[0] single thread time: 147.64517450332642
running on multi thread:
from threading import Thread
high_similarity = []
def measure(a = None,b= None,high_similarity = None):
d = ((a-b)**2).sum()
if d > 12:
high_similarity.append(d)
start_multi_thread = time.time()
for i in range(len(A)):
for j in range(len(B)):
if i<j:
thread = Thread(target=measure,kwargs= {'a':A[i],'b':B[j],'high_similarity':high_similarity} )
thread.start()
thread.join()
finish_multi_thread = time.time()
print("time to run on multi threads:",finish_multi_thread - start_multi_thread)
out[1] time to run on multi-threads: 11.946279764175415
A_array = np.array(A)
B_array = np.array(B)
start_vectorized = time.time()
for i in range(len(A_array)):
#vectorized distance operation
dists = (A_array-B_array)**2
high_similarity+= dists[dists>12].tolist()
aux = B_array[-1]
np.delete(B_array,-1)
np.insert(B_array, 0, aux)
finish_vectorized = time.time()
print("time to run vectorized operations:",finish_vectorized-start_vectorized)
out[2] time to run vectorized operations: 2.302949905395508
Note that you can't guarantee any order of execution, so will you also need to store the index of results. The snippet of code is just to illustrate that you can use parallel process, but i highly recommend to use a pool of threads and divide your dataset in N subsets for each worker and join the final result (instead of create a thread for each function call like i did).
I have a question about the performance of openpyxl when reading files.
I am trying to read the same xlsx file using ProcessPoolExecutor, single file Maybe 500,000 to 800,000 rows.
In read-only mode calling sheet.iter_rows(), when not using ProcessPoolExecutor, reading the entire worksheet, it takes about 1s to process 10,000 rows of data. But when I set the max_row and min_row parameters with ProcessPoolExecutor, it is different.
totalRows: 200,000
1 ~ 10000 take 1.03s
10001 ~ 20000 take 1.73s
20001 ~ 30000 take 2.41s
30001 ~ 40000 take 3.27s
40001 ~ 50000 take 4.06s
50001 ~ 60000 take 4.85s
60001 ~ 70000 take 5.93s
70001 ~ 80000 take 6.64s
80001 ~ 90000 take 7.72s
90001 ~ 100000 take 8.18s
100001 ~ 110000 take 9.42s
110001 ~ 120000 take 10.04s
120001 ~ 130000 take 10.61s
130001 ~ 140000 take 11.17s
140001 ~ 150000 take 11.52s
150001 ~ 160000 take 12.48s
160001 ~ 170000 take 12.52s
170001 ~ 180000 take 13.01s
180001 ~ 190000 take 13.25s
190001 ~ 200000 take 13.46s
total: take 33.54s
Obviously, just looking at the results of each process, the time consumed is indeed less.
But the overall time consumption has increased.
And the further back the scope, the more time each process consumes.
Read 200,000 rows with a single process only takes about 20s.
I'm not very clear with iterators and haven't looked closely at the source code of openpyxl.
From the time consumption, even if the range is set, the iterator still needs to start processing from row 1, I don't know if this is the case.
I'm not a professional programmer, if you happen to have relevant experience, please try to be as simple as possible
codes here!!!
import openpyxl
from time import perf_counter
from concurrent.futures import ProcessPoolExecutor
def read(file, minRow, maxRow):
start = perf_counter()
book = openpyxl.load_workbook(filename=file, read_only=True, keep_vba=False, data_only=True, keep_links=False)
sheet = book.worksheets[0]
val = [[cell.value for cell in row] for row in sheet.iter_rows(min_row=minRow, max_row=maxRow)]
book.close()
end = perf_counter()
print(f'{minRow} ~ {maxRow}', 'take {0:.2f}s'.format(end-start))
return val
def parallel(file: str, rowRanges: list[tuple]):
futures = []
with ProcessPoolExecutor(max_workers=6) as pool:
for minRow, maxRow in rowRanges:
futures.append(pool.submit(read, file, minRow, maxRow))
return futures
if __name__ == '__main__':
file = '200000.xlsx'
start = perf_counter()
tasks = getRowRanges(file)
parallel(file, tasks)
end = perf_counter()
print('total: take {0:.2f}s'.format(end-start))
Q :"... a question about the performance ..."... please try to be as simple as possible ...
A :Having 6 Ferrari sport racing cars ( ~ max_workers = 6 )does not provide a warranty to move 6 drivers ( ~ The Workload )
from start to the end
in 1 / 6 of the time.
That does not work,even if we have a 6-lane wide racing track ( which we have not ), as you have already reported, there is a bottleneck ( a 1-lane wide only bridge, on the way from the start to the end of the race ).
Actually,there are more performance-devastating bottlenecks ( The Bridge as the main performance blocker and a few smaller, less blocking, nevertheless performance further degrading bridges ), some avoidable, some not :
the file-I/O has been no faster than ~ 10k [rows/s] in a pure solo serial runso never expect the same speed to appear "across" the same (single, single lane) bridge ( the shared file-I/O hardware interface ) for any next, concurrently running Ferrari, competing for using the same resource, already used for the first process to read from file ( real-hardware latencies matter, a lot ... the Devil is in details )
another, avoidable, degradation comes with expensive add-on costs, paid for each and every list.append(). Here, try to choose a different object, avoiding a list-based storage at all and pre-allocate a block-storage ( one time paid RAM-allocation costs ) having an advantage of a know size of the result-storage, and keep storing data on-the-fly, best in cache-line respectful blocks than incrementally ( might be too technical, yet if performance is to get maxed-up, these details matter )
dual-iterator SLOC is nice for a workbook example, yet if performance is or focus, try to find another way, perhaps using even a simpler XLS-reader ( without as many machinery under the hood, as VBA interpreter et al ), which can export the row-wise consumed cells into a plain-text, that can get collected way way faster, than the as-is code did in a triplet-of-nested-iterators "syntax-sugared" SLOC [ [ ... for cell in row ] for row in sheet.iterator(...) ]
last comes also the process instantiation costs, that enter the revised Amdahl's Law, reformulated so that it takes into account also the overheads and atomicity of (blocks of) work. For ( technically independent ) details may see this and these - where interactive speedup-simulator calculators are often linked to test the principal ceiling any such parallelisation efforts will never be able to overcome.
Last, but by no means the least - The MEMORY: take your .xlsx file size and multiply it by ~ 50x and next by 6 workers ~ that amount of physical memory is expected to be used ( see doc: "Memory use is fairly high in comparison with other libraries and applications and is approximately 50 times the original file size, e.g. 2.5 GB for a 50 MB Excel file" credit to #Charlie Clark ) If your system does not have that much physical-RAM, the O/S starts to suffocate as truing to allocate that and goes into a RAM-swap-"thrashing" mode ( moving blocks-of-RAM to disk-swap area and back and there and back, as interleaving the 6 workers going forwards in Virtual-Memory-managed address space simulated inside a small physical-RAM at awfully high (more than 5(!) orders of magnitude longer) disk-I/O latencies, trying to cross the already blocking performance bottleneck, yeah - The Bridge ... where traffic-jam is already at max, as 6 workers try to do the very same - move some more data across the even more blocked bottleneck ) all that at awfully great latency skyrocketing jump on doing so (see URL on latencies above). A hint may, yet need not save us, plus this and this may reduce, better straight prevent further inefficiencies
I believe to have the same problem as OP.
The puzzling part is that once min_row and max_row is set on sheet.iter_rows(), concurrent execution does not apply anymore, as if there was some sort of global lock in effect.
The following code is trying to dump data from one single large sheet from an Excel file. The idea is to take advantage of the min_row and max_row on sheet.iter_rows to lock down a reading window and ThreadPoolExecutor for concurrent execution.
# artificially create a set of row index ranges,
# 10,000 rows per set till 1,000,000th row
# something like [(1, 10_000), (10_001, 20_000), .....]
def _ranges():
_i = 1
_n = 10_000
while _i <= 1_000_000:
yield _i, _i + _n - 1
_i += _n
def write_to_file(file, mn, mx):
print(f'write to file {mn}-{mx}')
wb = load_workbook(file, read_only=True
, data_only=True, keep_links=False, keep_vba=False)
sheet = wb[wb.sheetnames[0]]
out_file = _dst / f"{mn}-{mx}.txt"
row_count = 1
with out_file.open('w', encoding='utf8') as f:
rows = sheet.iter_rows(values_only=True, min_row=mn, max_row=mx)
for row in rows:
print(f'section {mn}-{mx} write {row_count}')
f.write(' '.join([str(c).replace('\n', ' ') for c in row]) + '\n')
row_count += 1
def main():
fut = []
with futures.ThreadPoolExecutor() as ex:
for mn, mx in _ranges():
fut.append(ex.submit(write_to_file, _file, mn, mx))
futures.wait(fut)
All write_to_file() do kick off all at once.
Iteration over rows, however, seems to behave in strict sequential fashion.
With a little change:
def write_to_file(file, mn, mx):
print(f'write to file {mn}-{mx}')
wb = load_workbook(file, read_only=True
, data_only=True, keep_links=False, keep_vba=False)
sheet = wb[wb.sheetnames[0]]
out_file = _dst / f"{mn}-{mx}.txt"
row_count = 1
with out_file.open('w', encoding='utf8') as f:
rows = sheet.iter_rows(values_only=True)
# ^^^^^^^^^^^^^^^^^___min_row/max_row not set
for row in rows:
print(f'section {mn}-{mx} write {row_count}')
f.write(' '.join([str(c).replace('\n', ' ') for c in row]) + '\n')
row_count += 1
Section 20001-30000 writes first!
The chaotic effect of concurrent execution takes place.
But, without min_row and max_row, there is no point to have concurrent execution at all.
I am currently parsing historic delay data from a public transport network in Sweden. I have ~5700 files (one from every 15 seconds) from the 27th of January containing momentary delay data for vehicles on active trips in the network. It's, unfortunately, a lot of overhead / duplicate data, so I want to parse out the relevant stuff to do visualizations on it.
However, when I try to parse and filter out the relevant delay data on a trip level using the script below it performs really slow. It has been running for over 1,5 hours now (on my 2019 Macbook Pro 15') and isn't finished yet.
How can I optimize / improve this python parser?
Or should I reduce the number of files, and i.e. the frequency of the data collection, for this task?
Thank you so much in advance. 💗
from google.transit import gtfs_realtime_pb2
import gzip
import os
import datetime
import csv
import numpy as np
directory = '../data/tripu/27/'
datapoints = np.zeros((0,3), int)
read_trips = set()
# Loop through all files in directory
for filename in os.listdir(directory)[::3]:
try:
# Uncompress and parse protobuff-file using gtfs_realtime_pb2
with gzip.open(directory + filename, 'rb') as file:
response = file.read()
feed = gtfs_realtime_pb2.FeedMessage()
feed.ParseFromString(response)
print("Filename: " + filename, "Total entities: " + str(len(feed.entity)))
for trip in feed.entity:
if trip.trip_update.trip.trip_id not in read_trips:
try:
if len(trip.trip_update.stop_time_update) == len(stopsOnTrip[trip.trip_update.trip.trip_id]):
print("\t","Adding delays for",len(trip.trip_update.stop_time_update),"stops, on trip_id",trip.trip_update.trip.trip_id)
for i, stop_time_update in enumerate(trip.trip_update.stop_time_update[:-1]):
# Store the delay data point (arrival difference of two ascending nodes)
delay = int(trip.trip_update.stop_time_update[i+1].arrival.delay-trip.trip_update.stop_time_update[i].arrival.delay)
# Store contextual metadata (timestamp and edgeID) for the unique delay data point
ts = int(trip.trip_update.stop_time_update[i+1].arrival.time)
key = int(str(trip.trip_update.stop_time_update[i].stop_id) + str(trip.trip_update.stop_time_update[i+1].stop_id))
# Append data to numpy array
datapoints = np.append(datapoints, np.array([[key,ts,delay]]), axis=0)
read_trips.add(trip.trip_update.trip.trip_id)
except KeyError:
continue
else:
continue
except OSError:
continue
I suspect the problem here is repeatedly calling np.append to add a new row to a numpy array. Because the size of a numpy array is fixed when it is created, np.append() must create a new array, which means that it has to copy the previous array. On each loop, the array is bigger and so all these copies add a quadratic factor to your execution time. This becomes significant when the array is quite big (which apparently it is in your application).
As an alternative, you could just create an ordinary Python list of tuples, and then if necessary convert that to a complete numpy array at the end.
That is (only the modified lines):
datapoints = []
# ...
datapoints.append((key,ts,delay))
# ...
npdata = np.array(datapoints, dtype=int)
I still think the parse routine is your bottleneck (even if it did come from Google), but all those '.'s were killing me! (And they do slow down performance somewhat.) Also, I converted your i, i+1 iterating to using two iterators zipping through the list of updates, this is a little more advanced style of working through a list. Plus the cur/next_update names helped me keep straight when you wanted to reference one vs. the other. Finally, I remove the trailing "else: continue", since you are at the end of the for loop anyway.
for trip in feed.entity:
this_trip_update = trip.trip_update
this_trip_id = this_trip_update.trip.trip_id
if this_trip_id not in read_trips:
try:
if len(this_trip_update.stop_time_update) == len(stopsOnTrip[this_trip_id]):
print("\t", "Adding delays for", len(this_trip_update.stop_time_update), "stops, on trip_id",
this_trip_id)
# create two iterators to walk through the list of updates
cur_updates = iter(this_trip_update.stop_time_update)
nxt_updates = iter(this_trip_update.stop_time_update)
# advance the nxt_updates iter so it is one ahead of cur_updates
next(nxt_updates)
for cur_update, next_update in zip(cur_updates, nxt_updates):
# Store the delay data point (arrival difference of two ascending nodes)
delay = int(nxt_update.arrival.delay - cur_update.arrival.delay)
# Store contextual metadata (timestamp and edgeID) for the unique delay data point
ts = int(next_update.arrival.time)
key = "{}/{}".format(cur_update.stop_id, next_update.stop_id)
# Append data to numpy array
datapoints = np.append(datapoints, np.array([[key, ts, delay]]), axis=0)
read_trips.add(this_trip_id)
except KeyError:
continue
This code should be equivalent to what you posted, and I don't really expect major performance gains either, but perhaps this will be more maintainable when you come back to look at it in 6 months.
(This probably is more appropriate for CodeReview, but I hardly ever go there.)
I have a catalog I opened in python, which has about 70,000 rows of data (ra, dec coordinates and object name) for various objects. I also have another list of about 15,000 objects of interest, which also appear in the previously mentioned catalog. For each of these 15,000 objects, I would like to see if any other objects in the large 70,000 list have ra, dec coordinates within 10 arcseconds of the object. If this is found to be true, I'd just like to flag the object and move on to the next one. However, this process takes a long time, since the distances are computed between the current object of interest (out of 15,000) 70,000 different times. This would take days! How could I accomplish the same task more efficiently? Below is my current code, where all_objects is a list of all the 15,000 object names of interest and catalog is the previously mentioned table data for 70,000 objects.
from astropy.coordinates import SkyCoord
from astropy import units as u
for obj_name in all_objects:
obj_ind = list(catalog['NAME']).index(obj_name)
c1 = SkyCoord(ra=catalog['RA'][obj_ind]*u.deg, dec=catalog['DEC'][obj_ind]*u.deg, frame='fk5')
for i in range(len(catalog['NAME'])):
if i != obj_ind:
# Compute distance between object and other source
c2 = SkyCoord(ra=catalog['RA'][i]*u.deg, dec=catalog['DEC'][i]*u.deg, frame='fk5')
sep = c1.separation(c2)
contamination_flag = False
if sep.arcsecond <= 10:
contamination_flag = True
print('CONTAMINATION FOUND')
break
1 Create your own separation function
This step is really easy once you look at the implementation and ask yourself: "how can I make this faster"
def separation(self, other):
from . import Angle
from .angle_utilities import angular_separation # I've put that in the code bellow so it is clearer
if not self.is_equivalent_frame(other):
try:
other = other.transform_to(self, merge_attributes=False)
except TypeError:
raise TypeError('Can only get separation to another SkyCoord '
'or a coordinate frame with data')
lon1 = self.spherical.lon
lat1 = self.spherical.lat
lon2 = other.spherical.lon
lat2 = other.spherical.lat
sdlon = np.sin(lon2 - lon1)
cdlon = np.cos(lon2 - lon1)
slat1 = np.sin(lat1)
slat2 = np.sin(lat2)
clat1 = np.cos(lat1)
clat2 = np.cos(lat2)
num1 = clat2 * sdlon
num2 = clat1 * slat2 - slat1 * clat2 * cdlon
denominator = slat1 * slat2 + clat1 * clat2 * cdlon
return Angle(np.arctan2(np.hypot(num1, num2), denominator), unit=u.degree)
It calculates a lot of cosines and sines, then creates an instance of Angle and converts to degrees then you convert to arc seconds.
You might not want to use Angle, nor do the tests and conversions at the beginning, nor doing the import in the function, nor doing so much variable assignment if you need performance.
The separation function feels a bit heavy to me, it should just take numbers and return a number.
2 Use a quad tree (requires a complete rewrite of your code)
That said, let's look at the complexity of your algorithm, it checks every element against every other element, complexity is O(n**2) (Big O notation). Can we do better...
YES You could use a Quad-tree, worst case complexity of Quad tree is O(N). What that basically means if you're not familiar with Big O is that for 15 000 element, the lookup will be 15 000 times what it is for 1 element instead of 225Â 000Â 000 times (15 000 squared)... quite an improvement right... Scipy has a great Quad tree library (I've always used my own).
really not long ago I had my first dumb question answered here so... there I am again, with a hopefully less dumb and more interesting headscratcher. Keep in my mind I am still making my baby steps in scripting !
There it is : I need to rig a feathered wing, and I already have all the feathers in place. I thought of mimicking another rig I animated recently that had the feathers point-constrained to the arm and forearm, and orient-constrained to three other controllers on the arm : each and every feather was constrained to two of those controllers at a time, and the constraint's weights would shift as you went down the forearm towards the wrist, so that one feather perfectly at mid-distance between the elbow and the forearm would be equally constrained by both controllers... you get the picture.
My reasoning was as follows : let's make a loop that iterates over every feather, gets its world position, finds the distance from that feather to each of the orient controllers (through Pythagoras), normalize that and feed the values into the weight attribute of an orient constraint. I could even go the extra mile and pass the normalized distance through a sine function to get a nice easing into the feathers' silhouette.
My pseudo-code is ugly and broken, but it's a try. My issues are inlined.
Second try !
It works now, but only on active object, instead of the whole selection. What could be happening ?
import maya.cmds as cmds
# find world space position of targets
base_pos = cmds.xform('base',q=1,ws=1,rp=1)
tip_pos = cmds.xform('tip',q=1,ws=1,rp=1)
def relative_dist_from_pos(pos, ref):
# vector substract to get relative pos
pos_from_ref = [m - n for m, n in zip(pos, ref)]
# pythagoras to get distance from vector
dist_from_ref = (pos_from_ref[0]**2 + pos_from_ref[1]**2 + pos_from_ref[2]**2)**.5
return dist_from_ref
def weight_from_dist(dist_from_base, dist_to_tip):
normalize_fac = (1/(dist_from_base + dist_to_tip))
dist_from_base *= normalize_fac
dist_to_tip *= normalize_fac
return dist_from_base, dist_to_tip
sel = cmds.ls(selection=True)
for obj in sel:
# find world space pos of feather
feather_pos = cmds.xform(obj, q=1, ws=1, rp=1)
# call relative_dist_from_pos
dist_from_base = relative_dist_from_pos(feather_pos, base_pos)
dist_to_tip = relative_dist_from_pos(feather_pos, tip_pos)
# normalize distances
weight_from_dist(dist_from_base, dist_to_tip)
# constrain the feather - weights are inverted
# because the smaller the distance, the stronger the constraint
cmds.orientConstraint('base', obj, w=dist_to_tip)
cmds.orientConstraint('tip', obj, w=dist_from_base)
There you are. Any pointers are appreciated.
Have a good night,
Hadriscus