Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 8 years ago.
Improve this question
I'm using Python in an attempt to analyse a large chunk of empiric measurements. In essence, I've two functions transforming the empiric data which also takes 3 'count' parameters - and returns a sequence of floats in each configuration. I'm expecting (hoping) to see some interesting patterns emerge when appropriate parameters are selected. I anticipate that the patterns might be relative between sequences returned for each function - and/or relate to patterns of some kind in the parameters. In case it's relevant, the 3 'count' parameters roughly correspond to:
A 'window size' on the underlying data over which summary statistics are calculated
A number of consecutive windows used to compute a single summary statistic (i.e. the trade-off between greater spatial or greater temporal accuracy)
An 'minimum age' - an offset into history of the underlying data.
The summary statistics (which generate the resulting sequences of floats for each parameter configuration) are non-trivial but will be independently sensitive to all three parameters.
I'm interested in visualisation techniques - suited to RAD/ad-hoc enquiry that will help me experiment with this multi-dimensional data.
So far, I've tinkered with MatPlotLib but find being restricted generating two graphs of 2/3 dimensions in the style of batch processing makes investigation very tedious. Ideally, I'd find a tool that would allow me to visualise more than two dimensions... perhaps allowing me to switch real-time between dimensions in an interactive GUI.
I'd really appreciate hints from any visualisation gurus as to suitable tools I should investigate - ideally to integrate with my existing Python functions - or in other languages. I'd especially like to hear any anecdotes of success with similar visualisation problems.
EDIT to add: One possible approach I'm considering is to use animation on 2 or 3D plots (to capture another dimension... leaving 1 or 2 for manual selection)... though I've found no good tools to help me achieve this, yet.
RGL is a visualization device system for R, using OpenGL as the rendering backend. An rgl device at its core is a real-time 3D engine written in C++. It provides an interactive viewpoint navigation facility (mouse + wheel support) and an R programming interface.
GGobi is an open source visualization program for exploring high-dimensional data. It provides highly dynamic and interactive graphics such as tours, as well as familiar graphics such as the scatterplot, barchart and parallel coordinates plots. Plots are interactive and linked with brushing and identification.
There's a tutorial that covers both of the above systems here.
RPy is a very simple, yet robust, Python interface to the R Programming Language. It can manage all kinds of R objects and can execute arbitrary R functions (including the graphic functions). All errors from the R language are converted to Python exceptions. Any module installed for the R system can be used from within Python.
You might want to look at outputting SVG with animation, in which case this question might interest you. I suspect the animation aspects will require a lot of work on your part. Another option is maybe visualizing the data as a graph, although I'm don't know enough about your data to know whether this would be useful to you. If it is, cytoscape is python scriptable
If all you want is an animated surface, then gnuplot can do it. A quick intro on it can be found here, or from the gnuplot FAQ. More detail can obviously be found in the gnuplot docs.
You could try guiqwt. It's aimed for 2D graphs, but targets more specifically interactive plots (as opposed to Matplotlib, although it can handle some degree of interaction too). From the guiqwt documentation:
Overview
Based on PyQwt (plotting widgets for PyQt4 graphical user interfaces)
and on the scientific modules NumPy and SciPy, guiqwt is a Python
library providing efficient 2D data-plotting features (curve/image
visualization and related tools) for interactive computing and
signal/image processing application development.
Performances
The most popular Python module for data plotting is currently
matplotlib, an open-source library providing a lot of plot types and
an API (the pylab interface) which is very close to MATLAB’s plotting
interface.
guiqwt plotting features are quite limited in terms of plot types
compared to matplotlib. However the currently implemented plot types
are much more efficient. For example, the guiqwt image showing
function (guiqwt.pyplot.imshow()) do not make any copy of the
displayed data, hence allowing to show images which are much larger
than with its matplotlib‘s counterpart. In other terms, when showing a
30-MB image (16-bits unsigned integers for example) with guiqwt, no
additional memory is wasted to display the image (except for the
offscreen image of course which depends on the window size) whereas
matplotlib takes more than 600-MB of additional memory (the original
array is duplicated four times using 64-bits float data types).
(I haven't tried it, so I can't comment on these claims.)
Okay, now that I understand your data I can definitely suggest a method of visualisation. A coloured 3D surface density plot. Use a0, a1 and a2 as standard x,y,z axes, use a3 as the time axis, and plot different colours over a monochromatic range (or cold to hot). That way the only thing that needs an interactive slider is a3.
As far as tools to do this are concerned
I don't know whether gnuplot can do colour density plots, if it can this is your best bet. Generate an set of gifs across domain of a3, use imagemagick to make a single animated gif out of them, then use an animated .gif editor that allows you to move back and forth between frames
Again, with matplotlib, I'm not certain whether it is possible to do colour density plots
SVG can definitely do everything you need to do, including the animation aspects, but as I've said before, is going to be a lot of hard work.
Sounds like Mayavi might fit your needs. It is written in Python, can be used interactively and supports 3D graphs and animations. You can have a look at this tutorial to see if it fits your needs.
I have done an interactive 3D visualization with animation in Python using the older version 1 of mayavi, see this page.
Edit
Unfortunately, most Mayavi examples show off too much advanced functionality. Here are two examples that demonstrate more basic applications. If these two do not fit your needs, then Mayavi may not be a good choice in your case. My understanding is that you have arrays of floats that you want to visualize.
Example 1
Here is a specific example from the older page on what you can do with a 3D array of floats: 3D data example. This example shows the use of isocontour surfaces, one solid cut plane through the data and another cut plane with isocontour lines. You can interactively move the cut planes around or choose different visualization tools. (In my case I had added another dimension and an animation that presented the data as 3D-cube slices through the hypercube.)
Example 2
Here is another example of what a more "conventional" plot with Mayavi could look like: Fourier transform example. This is quite similar to what the many other plotting libraries do.
Go download a free trial of Tableau (www.tableausofware.com). It will encode your data on X, Y, size, color and shape, and you can create small multiples any other dimensions you have -- i.e. you can look at lots of dimensions at once. You can try lots and lots of visualizations very rapidly. There is free training on the company website.
Disclaimer: I work for them.
The simplest visualization for 3+dimensions is bubble chart or motion chart. On top of the x and y axis you can use the bubble size and the bubble color for the extra dimensions.
Google visualization (http://code.google.com/apis/chart/interactive/docs/gallery/motionchart.html) and its google spreadsheet interactive mode give a simple interface to play with which of the dimensions is on which of the axis/size/color.
It is not aimed at handling too many data points, but you can use it to identify patterns on samples of the data with ease.
Related
My goal is to build a temperature gradient map over a floor plan to display minute changes in temp. via uniformly distributed sensors.
As far as I understand most heatmap tools available work with point density to produce heatmaps whereas what I'm looking for is a gradient based on varying values of individual points (sensors) on the map. I.e. something like this...
which I nicked from here.
From what I've gathered interpolation will definitely be involved and it may just be Radial Basis Function Interpolation because it wouldn't require a grid as per this post.
I've used the Anaconda distribution thus far. The data from sensors will be extracted from timescaleDB and the positions of sensors will be lat/long.
I've done very minor experimentation with the code from the link above and got this result. Radial Basis Function Interpolation
So here are my questions. Multiple python libraries have interpolation as a built-in function but which one of the libraries would be the best for the task described above? What parts of documentation should I read up on from libraries which can help me with this specific problem? Any good resource recommendations for this topic? Would anything else be required for this apart from interpolation?
Thanks in advance!
P.S. This is a side project I'd like to work on as a student, not commercial in any way shape or form.
I like scipy.interpolate library. It has a lot of nice functions, the simplest that would work for you would probably be the scipy.interpolate.interp2d(), and if you want to go with an non linear distribution of sensors griddata() is very useful.
I have a series of points (x,y,z) that I would like to plot as a vector in a 3D plane. Something like this.
I am successfully using QCustomPlot elsewhere, but the documentation says it cannot be used for 3D plots. Googling turned up QwtPlot3D, but it hasn't been maintained since 2007, as far as I can tell, and I don't want to run into any problems since I'm using Qt5. I was also looking at QtCharts but can't seem to find any example of plotting x,y,z data points.
Does anyone have tips for including a 3D graph in my C++/Qt application? Is there a tool that would work better with Python with Qt, rather than C++? Or another technology entirely? This graph will be part of a larger UI.
This might help, though I haven't used it:
http://doc.qt.io/QtDataVisualization/
Spend a little time looking into OpenGL. To display OpenGL scenes in Qt you would use QGLWidget (for Qt 4.x) or QOpenGLWidget (for Qt 5.x). OpenGL allows you to write graphics that run on a GPU card, meaning you can tap into the same horsepower used for 3D video games. Given time and inclination, you can build up a good 3D graphics library.
https://www.opengl.org/
http://doc.qt.io/qt-5/qopenglwidget.html
The Qt tutorials can help, but you'll also want to read other OpenGL tutorials. Here are some tutorials targeting older versions of Qt:
ftp://ftp.informatik.hu-berlin.de/pub1/Mirrors/ftp.troll.no/QT/pub/developerguides/qtopengltutorial/OpenGLTutorial.pdf
http://www.decom.ufop.br/sibgrapi2012/eproceedings/tutorials/t3-survey_paper.pdf
Tutorials tend to start with "immediate mode" examples, meaning the CPU is continually involved with updating data and writing that data to the GPU. As soon as you grasp the basics you'll want to implement "retained mode" code, meaning (very loosely) that the GPU manages the data and the need for CPU resources is minimized.
All that said, getting into OpenGL is a commitment. If you want the user to be able to change the viewpoint of the chart, or zoom in/out, or mouse over a plot to check individual values, etc., then it will take some time to implement. For a standard that's so widely use it's odd that the documentation and available textbooks aren't better--don't expect to find the OpenGL textbook equivalent of Kernighan & Ritchie or the Perl camel book.
There may be some Qt 3D graphing project somewhere that enjoys active development, and with luck maybe some other SO user will know about one.
I am a meteorologist, and lately I am trying to investigate the possibility of building my one sondes.
In order to do that, I have the following work plan :
I would like to generate 3D models pyformex. An alternative is openSCAD. But I start with pyformex - to generate simple cylindrical sonde shapes with associated extra features, e.g. intake tube or such.
Next, I will like to split it in Meshes, using PyDistMesh; as well as prepare a raytraced point cloud model with Xrt.
In the third step, I would like to perform the CFD works.
Now, my questions :
Are there some other simple Python Libraries to generate 3D models? I would like a very simple system, where i can issue commands like p = Parallelogram (length, height, width), or p.position(x,y,z) etc. It would be nice to have built in mouse interaction - that is, a built in drawing component, which I can use to display the model, and rotate/ zoom/pan with mouse.
Any other mesh generation tools?
For this step, I would need a multiphysics system. I tried to use OpenFOAM, it is too huge (to hack through). I have taken a look at SU2, but it seems to focus more on aerospace engineering, than Fluid Dynamics (I would like to simulate the flight of the sonde - which is closer to aerospace engineering, as well as the state of the atmosphere). Fluidity seems to suit my needs better, but I dont find a python fork thereof. So are there some general purpose, not too bloated up, multiphysics python library for geophysical and general hydrodynamic simulations? I have taken a look a MOOSE, also dont find a python binding for it.
Scientific visualization : Are there some 3 or 4 (or may be higher dimensional) visualization libraries? I would prefer to issue simple commands as Plot instead of first generating a window / form, and then putting the graphs on it, if possible.
FINALLY, and most importantly, if the same can be done by C++ or Fortan, or some other language besides java, I would also consider using those.
Have a look at http://freecadweb.org/. This seems to be under active development. It is a fairly complete open source CAD package written in python. I believe it also has tools for meshing.
For cfd, you might want to consider openfoam - http://www.openfoam.com/. This is an open source cfd package with the obligatory steep learning curve. There seem to be some python libraries to be available that link to it, however I'm not sure how active these are:
http://openfoamwiki.net/index.php/Contrib/PyFoam
http://pythonflu.wikidot.com/
what function should i use to draw the above performance profile of different algorithms, the running time data is from python implementation, stored in lists for different algorithm. Is there any build-in function in Python or Matlab to draw this kind of figure automatically? Thanks
You will have to store the performance of your algorithms in a list (if you are using Python) or in a matrix (Matlab) and then plot the results.
Matlab comes with several plot functions such as figure(), plot(), subplot() that you will need to draw those figures.
In Python you can use the third-party package Matplotlib. By the way, Matplotlib gives you a MatLab like environment called PyLab that tries to emulate Matlab commands.
You can make something like that in Python with Matplotlib using step and plot.
In Matlab you can use plot and stairs.
A Python function that draws these profiles which can be found, with instructions, here. After importing this function and generating a Pandas DataFrame object using your data (see instructions), the generation of the performance profile is easy.
A MATLAB script that draws performance profiles can be found here.
I make simple performance charts, like those posed in the question, via Python scripts that rely on a Python module named leather. For example, within the following gist.github.com repository:
https://gist.github.com/lhuemill/f1273291e5f5e85e4b42e5c7614e60ef
is a Python script, named permutations2x2_chart_results, which does this. By default, this script produces a chart using the SVG format. A format that is directly supported by most of the popular web browsers and that are a vector versus bitmap graphic. Vector graphics tend to be a good choice for these types of performance graphs, in that the user can zoom into any portion of the results, to obtain more detail, without pixelation.
I took a scientific programming course this semester that I really enjoyed and experimented with a lot. We used python, and all the related modules. I am taking a physics lab next semester and I just wanted to hear from some of you how python can help me in ways that excel can't or in ways that are better than excel's capabilities. I use Mathematica for symbolic stuff so I would use python for data purposes.
Off the top of my head, here are the related things I can do:
All of the things you would expect in a intro course (loops, arrays, slicing arrays, etc).
Reading data from a text file.
Plotting scatter, line, and bar graphs.
Learning how to plot linear regression but haven't totally figured it out.
I have done 7 of the problems on Project Euler (nothing to brag about, but it might give you a better idea of where I stand in skills).
Looking forward to hearing from some of you. You don't have to explain how to use the things you mention, I could look up the documentation.
The paper Python all a scientist needs comes to mind. I hope you can make the needed transformations from Biology to Physics.
Scipy will also be useful to you, as it includes many more advanced analysis tools. For example, Scipy includes a linear regression, and gets more interesting from there. Along with the other tools you mentioned, you'll probably find most of your needs covered.
Other notes on tool selection:
Mathematica is a great tool, if you can afford it. I've played around with the other options, like Sympy, and sadly, they don't come close to being as useful as Mathematica.
I can't imagine using Excel for any serious scientific work. If you're planning to continue forward using the tools that you learn in class, you might as well start with tools that offer you that potential.
Don't reject Excel outright. It's still great for doing simple data analysis and plotting. Excel also has the considerable advantage of being installed on most engineer and scientist's computers, making it a lot easier to share your work with colleagues.
That said, I do use Python when Excel just won't cut it; times when I've had to:
color the points in a scatter plot based on a third column
plot a field of vectors
extract a few values from each of several thousand data files to do statistical process control
generate dozens of scatter plots over different dimensions of a large data set to find which variables are important
solve a nonlinear equation at several intermediate points of a calculation, not just as the final result.
accept variable length input from a user to define a problem
VBA in Excel can do a lot of those things too, but it becomes painful fast in such a primitive language. I dream that Microsoft will make IronPython a first-class scripting language in the next version of Excel. Until then, you might want to try Resolver One
I can recall 2 presentations by Jan Martinek on EuroScipy 2008, he's PhD candidate and presented some fun experiments with Physics in the background. Abstracts are here and I'm sure he would't mind to share more if you contact him directly. Also, take a look at other presentation from EuroScipy, there are some more Physics-related ones.