Develop Reference

scipy.spatial.transform.Rotation Array of rotation vs Stack inside the same object

In a project I have to store a sequence of successive rotations of an object in python. I intend to store them using scipy.spatial.transform.Rotation objects.
I have noticed that two options are available.
I store a multitude of Rotation object in an array, each Rotation object containing a single rotation of my sequence.
I store every rotation in the same Rotation object, effectively stacking them in a single object.
I wondered what were the trade offs between the methods in terms of computational speed and data access speed. And ultimately which method should be preferred.
In my particular case:
I have a fairly big set of rotations (around 2'000'000).
I would like to speed up the accessing time for my data as much as possible.
I am not concerned with memory usage so I am willing to trade space for speed.
I don't intend to apply my rotations to anything. This is purely for storage use.
I will have to access my rotations both as quaternions (.as_quat()) and as euler angles (.as_euler())
I will have to split my data into smaller chunk at some point (And I am aware that if I use a single Rotation object I might have to re-create on per chunk in order to split my data correctly)
What I am looking for:
The method that has the fastest access time
The method that is closer to good practice and coding convention in python
Thanks in advance.

Alright so I'll throw in the first elements I have.
For both you end up accessing an iterable and an object (just in a different order). Therefor there should not be any major difference in accessing speed.
The list of R is easier to access and manipulate afterwards. Hence if you are suceptible of changing anything in your rotation sequence it is the easier way. There for using a single object requires some extra-code for manipulation and might be slower.
However it really depends on what you do and I have no data to prove that it is significantly slower.
The single object should take less memory space since there is only one instance, where as the list of object has nb_rotation times more.
I already mentionned that this was not critical in my case it is not critical. Again I don't have any data to support this but I expect the difference to be significative since I have about 2'000'000 rotations.
Regarding those facts I would make the decision as follow:
If you're just looking to use the object for storage then the single object is probably more efficient. (However that would apply very specific cases since you could also compute the resulting rotation and store a single quaternion or rotation object)
If you're looking to manipulate your data in any way then the list of object seems more indicated for that. The only real draw back is memory use how ever if you manipulate the data you would also have to unpack the single object at some point, leading to a similar problem, if not worse due to data duplication.
In my case I will be using the list of object.

Numba-compatible replacement for dict

I am trying to solve a series of Bellman value functions without using dictionaries, since I believe that dictionaries are not supported in Numba (e.g. see Optimizing dict of set of tuple of ints with Numba?). In particular, the dictionary structure in my naive representation (below) is useful because I need to perform operations on both the keys as well as the values. What data structures should I use, or how should I set up my problem so that I can still use Numba?
Explicitly: I am filling in a set of recursively defined value functions, where they are meant to model 'job experience' over various 'sectors' at a given time. For example, if there are 2 sectors and it is t=3, the experience allocations, i.e. the possible states at t=3, are (3,0), (2,1), (1,2), (0,3). For each of these experience allocations there is some value associated with them. The recursion comes from the fact that at t=2, the value can be recursively defined from the values of 'reachable' states at t=3. For example, at t=2, the value of (2,0) is a function of two values at t=3, (3,0) and (2,1), since those are the 'reachable' experience levels by adding 1 to experience in either sector.
This is very straightforward to code if I define a dictionary with keys that are tuples of form (time, experience), since I can just 'read' the experience (tuple), find out what are the 'reachable' key values in the next period (by adding 1 to each value of the experience tuple), call the corresponding values, and evaluate the value at the current time using the recursive relationship.
However, I do not believe dictionaries work in Numba, so what sort of data structure should I use instead, that still lets me pull in a 'key' and a 'value'? In general the problem will be too large for me to make e.g. a 2-D array of size N^T for N sectors for time T. Also, for my desired 'output', I want the full 'dictionary' of values, since I will presumably calculate this filled in dictionary once and then re-using its values numerous times.

How do I work with large, memory hungry numpy array?

I have a program which creates an array:
List1 = zeros((x, y), dtype=complex_)
Currently I am using x = 500 and y = 1000000.
I will initialize the first column of the list by some formula. Then the subsequent columns will calculate their own values based on the preceding column.
After the list is completely filled, I will then display this multidimensional array using imshow().
The size of each value (item) in the list is 24 bytes.
A sample value from the code is: 4.63829355451e-32
When I run the code with y = 10000000, it takes up too much RAM and the system stops the run. How do I solve this problem? Is there a way to save my RAM while still being able to process the list using imshow() easily? Also, how large a list can imshow() display?

There's no way to solve this problem (in any general way).
Computers (as commonly understood) have a limited amount of RAM, and they require elements to be in RAM in order to operate on them.
An complex128 array size of 10000000x500 would require around 74GiB to store. You'll need to somehow reduce the amount of data you're processing if you hope to use a regular computer to do it (as opposed to a supercomputer).
A common technique is partitioning your data and processing each partition individually (possibly on multiple computers). Depending on the problem you're trying to solve, there may be special data structures that you can use to reduce the amount of memory needed to represent the data - a good example is a sparse matrix.
It's very unusual to need this much memory - make sure to carefully consider if it's actually needed before you dwell into the extremely complex workarounds.

Which data structure is appropriate for this?

I have a line in my code that currently does this at each step x:
myList = [(lo,hi) for lo,hi in myList if lo <= x <= hi]
This is pretty slow. Is there a more efficient way to eliminate things from a list that don't contain a given x?

Perhaps you're looking for an interval tree. From Wikipedia:
In computer science, an interval tree is an ordered tree data structure to hold intervals. Specifically, it allows one to efficiently find all intervals that overlap with any given interval or point.
So, instead of storing the (lo, hi) pairs sequentially in a list, you would have them define the intervals in an interval tree. Then you could perform queries on the tree with x, and retain only the intervals that overlap x.

While you don't give much context, I'll assume the rest of the loop looks like:
for x in xlist:
myList = [(lo,hi) for lo,hi in myList if lo <= x <= hi]
In this case, if may be more efficient to construct an interval tree (http://en.wikipedia.org/wiki/Interval_tree) first. Then, for each x you walk the tree and find all intervals which intersect with x; add these intervals to a set as you find them.

Here I'm going to suggest what may seem like a really dumb solution favoring micro-optimizations over algorithmic ones. It'll depend on your specific needs.
The ultimate question is this: is a single linear pass over your array (list in Python), on average, expensive? In other words, is searching for lo/high pairs that contain x going to generally yield results that are very small (ex: 1% of the overall size of the list), or relatively quite large (ex: 25% or more of the original list)?
If the answer is the latter, you might actually get a more efficient solution keeping a basic, contiguous, cache-friendly representation that you're accessing sequentially. The hardware cache excels at plowing through contiguous data where multiple adjacent elements fit into a cache line sequentially.
What you want to avoid in such a case is the expensive linear-time removal from the middle of the array as well as possibly the construction of a new one. If you trigger a linear-time operation for every single individual element you remove from the array, then naturally that's going to get very expensive very quickly.
To exchange that linear-time operation for a much faster constant-time one, all we have to do when we want to remove an element at a certain index in the array is to overwrite the element at that index with the element at the back of the array (last element). Now simply remove the redundant duplicate from the back of the array (a removal from the back of an array is a constant-time operation, and often involves just basic arithmetical instructions).
If your needs fit the criteria, then this can actually give you better results than a smarter algorithm. It's one of the peculiar cases where the practice can trump the theory due to the skewed performance of the hardware cache over DRAM, but if you're performing these types of hi/lo queries repeatedly and wanting to get very narrow results, then something smarter like an interval tree or at least sorting the data to allow binary searches can be considerably better.

Find sublist in list with different scaling / fuzzy pattern recognition

I have two lists, each containing an ordered set of numbers.
One list is small (~ 5 - 20 elements) the other one is large (~ 5000). The lists have a different "scaling" and there might be points missing in one or the other list. In general most elements will be in both lists.
I'm looking for a method to detect the position and the "scaling" between the two lists, such that the distance between the two lists has a minimum.
An example would be:
l1 = [ 100., 200., 400.]
l2 = [ 350., 1000., 2003., 3996., 7500., 23000.]
The scale would be 10. and the position of l1 in l2 is 1.
The list 10.*l1 appears at position 1 within l2; the lists have a distance of 7 (this depends on the metric I choose, here I just summed up the differences between all elements).
I'm wondering if there are already methods out there e.g. in pattern recognition which I can use (preferably in python). It seems to me that this could be a common problem when comparing patterns with unknown scaling factors. But I couldn't find a good keyword which describes my problem.
The application of this is to identify measured spectroscopic lines by comparing them to a catalog of the positions of known lines and therefore converting the unphysical unit "pixel on the detector" to actual wavelength.
In principle I could already provide a decent guess of the scaling factor of the two lists, but I guess this will not be necessary, as the solutions should be unique in most cases.
any help is appreciated,
Julian

The problem you're trying to solve is an optimization in two degrees. The first being the scale and the second being the index. The broad sense of your problem is generally difficult to solve efficiently. However there are a few things that could simplify the calculations. First are both sets sorted? Second are you looking for the consecutive set from the second list that matches the first or not? To explain that further I'll use an example: 1, 2, 3 and 2, 3, 4, 6. is the scale better as 2 (skipping the 3 in the second list) or 1.something (not skipping the 3)? Third what is the weighting you want to use to measure the difference between the two (linear sum, root mean square, etc.)?
If you can provide some of these details I may be able to give you a better idea of some things to try.
UPDATE
So based on your comment you can skip values. That actually makes this problem very difficult to solve O(2^n). Because you are basically looking at all combinations of list one with list two.
Even though you can optimize some aspects of this problem because they are sorted you will still have to check a lot of combinations.