Develop Reference

Python - Oscilloscope

I am trying to create a program in python which can create an oscilloscope from live audio (through a microphone)
The difference to a normal oscilloscope is that this will only show one wave length, for example (desired output):
This shows three different wavelengths, and how they will each show on the program.
My progress so far:
I've created a program to show a graph and clear and redraw it
I've then created a program which will display the sound live (Although it's very slow, which would ideally be fixed if possible)
Code #1:
import matplotlib.pyplot as plt
import time
plt.ion()
#y1 is the data
y1 = [0,0.309,0.587,0.809,0.951,1,0.951,0.809,0.587,0.309,0, -0.309, -0.587, -0.809, -0.951, -1, -0.951, -0.809, -0.587, -0.309, 0]
plt.plot(y1, 'r.-') #Graph with data
plt.plot([0 for _ in y1]) #Straight line at y=0
while True:
#Update data to new data
#y1 = new data
plt.plot(y1, 'r.-') #Graph with data
plt.plot([0 for _ in y1]) #Straight line at y=0
plt.draw()
plt.pause(0.5) #Time for one wave? Need some way to find this...
plt.clf()
Code #2:
import pyaudio
import numpy as np
import matplotlib.pyplot as plt
import time
RATE = 44100
CHUNK = int(RATE/20) # RATE / number of updates per second
def soundplot(stream):
t1=time.time()
data = np.fromstring(stream.read(CHUNK),dtype=np.int16)
plt.pause(0.1) #To draw graph!
plt.clf()
plt.plot(data)
plt.draw()
plt.axis([0,len(data),-2**16/2,2**16/2])
print("took %.02f ms"%((time.time()-t1)*1000))
if __name__=="__main__":
p=pyaudio.PyAudio()
stream=p.open(format=pyaudio.paInt16,channels=1,rate=RATE,input=True,
frames_per_buffer=CHUNK)
for i in range(int(20*RATE/CHUNK)): #do this for 10 seconds
soundplot(stream)
stream.stop_stream()
stream.close()
p.terminate()
Edit: To make it clear, my desired outcome is to show one single wave length as seen in the picture, instead of multiple which is what the second code produces

To show a single wavelength, first scan forward in time until you find a non-negative data point. (This would happen at once, with the very first entry, with the data in Code #1.)
Then continue scanning forward in time, and maintain a delta between successive samples. Initially the delta (or discrete derivative) will be positive as the curve approaches its max, then will turn negative till hitting a min, then will turn positive again.
Stop scanning forward in time when you encounter a non-negative data point and the delta is positive. At that point you have a full wavelength.
EDIT:
If you have lots of data then it's OK to skip some preamble data samples. The key here is we want to find a zero crossing with positive derivative, and then keep going until we find another zero crossing with positive derivative. So that first decision should both look for a non-negative data point and insist on positive delta.
In the presence of noise, we might see sign changes on delta more often than the period of the waveform. So an initial step might be to find min and max values for a bunch of samples (implying a range), then pick arbitrary thresholds like min + .25 * range and min + .75 * range, record the first positive zero crossing, wait for signal to exceed the high threshold, wait for it to go below the low threshold (after a negative zero crossing), then record the next positive zero crossing. That gives you an estimate of wavelength. Make repeated estimates if you find that helpful, and take some convenient aggregate like the mean or (better) the median.
Armed with a wavelength estimate, you're in a better position to evaluate whether a pair of positive zero crossings appear to be "correct" or due to noise. Reject pairs that are much closer together than your estimate would suggest. You may also find it convenient to compute a smoothed derivative, so instead of delta of last two points seen (K=2), you are averaging over the last K points, perhaps half a dozen of them. The average function is a low pass filter which rejects high frequency noise.

fft power spectrum woes

I'm having trouble getting a frequency spectrum out of a fourier transform... I have some data:
That I have mean-centered, and doesn't seem to have too much of a trend...
I plot the fourier transform of it:
And I get something that is not nice....
Here is my code:
def fourier_spectrum(X, sample_freq=1):
ps = np.abs(np.fft.fft(X))**2
freqs = np.fft.fftfreq(X.size, sample_freq)
idx = np.argsort(freqs)
plt.plot(freqs[idx], ps[idx])
As adapted from code taken from here.
It seems to work for some naive sin wave data:
fourier_spectrum(np.sin(2*np.pi*np.linspace(-10,10,400)), 20./400)
So my questions are: I'm expecting a non-zero-almost-everywhere-spectrum, what am I doing wrong? If I'm not doing anything wrong, what features of my data are causing this? Also, if I'm not doing anything wrong, and fft is just unsuited for my data for some reason, what should I do to extract important frequencies from my data?

It turns out that I just didn't understand the units of the x-axis in the frequency spectrum, which is Hz. Because my sample spacings were on the order of a second, and my period was on the order of a day, the only units really visible on my frequency spectrum were ~1/s (at the edges) to about ~1/m (near the middle), and anything with a longer period than that was indistinguishable from 0. My misunderstanding stemmed from the graph on this tutorial, where they do conversions so that the x-axis units are in time, as opposed to inverse time. I rewrote my frequency_spectrum plotting function to do the appropriate "zooming" on the resulting graph...
def fourier_spectrum(X, sample_spacing_in_s=1, min_period_in_s=5):
'''
X: is our data
sample_spacing_in_s: is the time spacing between samples
min_period_in_s: is the minimum period we want to show up in our
graph... this is handy because if our sample spacing is
small compared to the periods in our data, then our spikes
will all cluster near 0 (the infinite period) and we can't
see them. E.g. if you want to see periods on the order of
days, set min_period_in_s=5*60*60 #5 hours
'''
ps = np.abs(np.fft.fft(X))**2
freqs = np.fft.fftfreq(X.size, sample_spacing_in_s)
idx = np.argsort(freqs)
plt.plot(freqs[idx], ps[idx])
plt.xlim(-1./min_period_in_s,1./min_period_in_s) # the x-axis is in Hz

FFT using Python - unexpected low frequencies

I'm still trying to get frequency analysis for this data using FFT in Python.
The sampling rate is 1 data point per minute.
My code is:
from scipy.fftpack import fft
df3 = pd.read_csv('Pressure - Dates by Minute.csv', sep=",", skiprows=0)
df3['Pressure FFT'] = df3['ATMOSPHERIC PRESSURE (hPa) mean'] - df3['ATMOSPHERIC PRESSURE (hPa) mean'].mean()
Pressure = df3['Pressure FFT']
Fs = 1/60
Ts = 1.0/Fs
n = len(Pressure)
k = np.arange(n)
T = n/Fs
t = np.arange(0,1,1/n) # time vector
frq = k/T # two sides frequency range
frq = frq[range(int(n/2))] # one side frequency range
Y = np.fft.fft(Pressure)/n # fft computing and normalization
Y = Y[range(int(n/2))]
fig, ax = plt.subplots(2, 1)
ax[0].plot(t,Pressure)
ax[0].set_xlabel('Time')
ax[0].set_ylabel('Amplitude')
ax[1].plot(frq,abs(Y),'r') # plotting the spectrum
ax[1].set_xlabel('Freq (Hz)')
ax[1].set_ylabel('|Y(freq)|')
But the result gives:
So my problems are:
1) Why there are no frequencies at all ? The data is clearly periodic.
2) Why the frequency spectrum is so low ? (0 - 0.009)
3) Maybe I should try different filtering technique?
Any insights ?
Thanks !!!

1) Why there are no frequencies at all ? The data is clearly periodic.
Well, there is frequency content, it's just not exactly visible because of its structure. Try changing the line that plots the frequency spectrum, from ax[1].plot(frq,abs(Y),'r') to ax[1].semilogy(frq,abs(Y),'r')
This will result to:
Where we have now applied a simple transformation that boosts low values and limits high values. For more information please see this link. Of course, having removed the DC (as you do on line 3 of your code) helps too.
This still seems a bit blurry and it is, but if we zoom in to the lower part of the spectrum, we see this:
Which shows a spike at approximately 2.3e-05 Hz which corresponds to approximately 12 hours.
2) Why the frequency spectrum is so low ? (0 - 0.009)
Because you sample once every 60 seconds, therefore your sampling frequency is (approximately) 0.016 Hz. Your spectrum contains everything between DC (0Hz) and 0.0083Hz. For more information, please see this link
3) Maybe I should try different filtering technique?
You can try windowing if you can't resolve a harmonic but it doesn't look like it's needed here.
Hope this helps.

Part of the reason why those frequencies seem so low is because the time axis in your amplitude plot is scaled weirdly. If you really have one sample per 60 seconds then the x-axis should range between 0 and 1690260 seconds (i.e. ~20 days!).
By eye, you seem to have about one small peak every 50000 seconds (~2 per day), which would correspond to a frequency of about 2x10⁻⁵ Hz. Your periodogram therefore looks pretty reasonable to me, given how massive the scale of the x-axis is.

numpy Fourier transformation produces unexpected results

I am currently learning Fourier transformation and using Python to play around with it.
I have a code snippet here:
x = np.arange(0,50,0.1)
T = 5
y = np.sin(T*np.pi*x)
freq = np.fft.rfftfreq(x.size)
y_ft = np.fft.rfft(y)
plt.plot(freq, np.abs(y_ft))
It produces a correct chart as following:
But when I change the T into 10, the chart is like this:
I was expecting that I will get a similar chart like the first one with a right shift of the peak, because I just enlarged the cycle time.
Why increasing the cycle time would produces such an unexpected result?

You are effectively sampling a signal. With your code, the frequency you are sampling at is 1/0.1 or 10 rad/second. The frequency of your first sinusoid is just on the Nyquist frequency (5 rad/second). The frequency of your second sinusoid is beyond Nyquist, therefore your signal is not correctly sampled. Solution: increase your sampling frequency (x = np.arange(0, 50, 0.01) for example).
Look at what your T=10 signal looks like when plotted (you can see it doesn't resemble a single sinusoid at the sampling points):

How to correlate two time series with gaps and different time bases?

I have two time series of 3D accelerometer data that have different time bases (clocks started at different times, with some very slight creep during the sampling time), as well as containing many gaps of different size (due to delays associated with writing to separate flash devices).
The accelerometers I'm using are the inexpensive GCDC X250-2. I'm running the accelerometers at their highest gain, so the data has a significant noise floor.
The time series each have about 2 million data points (over an hour at 512 samples/sec), and contain about 500 events of interest, where a typical event spans 100-150 samples (200-300 ms each). Many of these events are affected by data outages during flash writes.
So, the data isn't pristine, and isn't even very pretty. But my eyeball inspection shows it clearly contains the information I'm interested in. (I can post plots, if needed.)
The accelerometers are in similar environments but are only moderately coupled, meaning that I can tell by eye which events match from each accelerometer, but I have been unsuccessful so far doing so in software. Due to physical limitations, the devices are also mounted in different orientations, where the axes don't match, but they are as close to orthogonal as I could make them. So, for example, for 3-axis accelerometers A & B, +Ax maps to -By (up-down), +Az maps to -Bx (left-right), and +Ay maps to -Bz (front-back).
My initial goal is to correlate shock events on the vertical axis, though I would eventually like to a) automatically discover the axis mapping, b) correlate activity on the mapped aces, and c) extract behavior differences between the two accelerometers (such as twisting or flexing).
The nature of the times series data makes Python's numpy.correlate() unusable. I've also looked at R's Zoo package, but have made no headway with it. I've looked to different fields of signal analysis for help, but I've made no progress.
Anyone have any clues for what I can do, or approaches I should research?
Update 28 Feb 2011: Added some plots here showing examples of the data.

My interpretation of your question: Given two very long, noisy time series, find a shift of one that matches large 'bumps' in one signal to large bumps in the other signal.
My suggestion: interpolate the data so it's uniformly spaced, rectify and smooth the data (assuming the phase of the fast oscillations is uninteresting), and do a one-point-at-a-time cross correlation (assuming a small shift will line up the data).
import numpy
from scipy.ndimage import gaussian_filter
"""
sig1 and sig 2 are assumed to be large, 1D numpy arrays
sig1 is sampled at times t1, sig2 is sampled at times t2
t_start, t_end, is your desired sampling interval
t_len is your desired number of measurements
"""
t = numpy.linspace(t_start, t_end, t_len)
sig1 = numpy.interp(t, t1, sig1)
sig2 = numpy.interp(t, t2, sig2)
#Now sig1 and sig2 are sampled at the same points.
"""
Rectify and smooth, so 'peaks' will stand out.
This makes big assumptions about your data;
these assumptions seem true-ish based on your plots.
"""
sigma = 10 #Tune this parameter to get the right smoothing
sig1, sig2 = abs(sig1), abs(sig2)
sig1, sig2 = gaussian_filter(sig1, sigma), gaussian_filter(sig2, sigma)
"""
Now sig1 and sig2 should look smoothly varying, with humps at each 'event'.
Hopefully we can search a small range of shifts to find the maximum of the
cross-correlation. This assumes your data are *nearly* lined up already.
"""
max_xc = 0
best_shift = 0
for shift in range(-10, 10): #Tune this search range
xc = (numpy.roll(sig1, shift) * sig2).sum()
if xc > max_xc:
max_xc = xc
best_shift = shift
print 'Best shift:', best_shift
"""
If best_shift is at the edges of your search range,
you should expand the search range.
"""

If the data contains gaps of unknown sizes that are different in each time series, then I would give up on trying to correlate entire sequences, and instead try cross correlating pairs of short windows on each time series, say overlapping windows twice the length of a typical event (300 samples long). Find potential high cross correlation matches across all possibilities, and then impose a sequential ordering constraint on the potential matches to get sequences of matched windows.
From there you have smaller problems that are easier to analyze.

This isn't a technical answer, but it might help you come up with one:
Convert the plot to an image, and stick it into a decent image program like gimp or photoshop
break the plots into discrete images whenever there's a gap
put the first series of plots in a horizontal line
put the second series in a horizontal line right underneath it
visually identify the first correlated event
if the two events are not lined up vertically:
select whichever instance is further to the left and everything to the right of it on that row
drag those things to the right until they line up
This is pretty much how an audio editor works, so you if you converted it into a simple audio format like an uncompressed WAV file, you could manipulate it directly in something like Audacity. (It'll sound horrible, of course, but you'll be able to move the data plots around pretty easily.)
Actually, audacity has a scripting language called nyquist, too, so if you don't need the program to detect the correlations (or you're at least willing to defer that step for the time being) you could probably use some combination of audacity's markers and nyquist to automate the alignment and export the clean data in your format of choice once you tag the correlation points.

My guess is, you'll have to manually build an offset table that aligns the "matches" between the series. Below is an example of a way to get those matches. The idea is to shift the data left-right until it lines up and then adjust the scale until it "matches". Give it a try.
library(rpanel)
#Generate the x1 and x2 data
n1 <- rnorm(500)
n2 <- rnorm(200)
x1 <- c(n1, rep(0,100), n2, rep(0,150))
x2 <- c(rep(0,50), 2*n1, rep(0,150), 3*n2, rep(0,50))
#Build the panel function that will draw/update the graph
lvm.draw <- function(panel) {
plot(x=(1:length(panel$dat3))+panel$off, y=panel$dat3, ylim=panel$dat1, xlab="", ylab="y", main=paste("Alignment Graph Offset = ", panel$off, " Scale = ", panel$sca, sep=""), typ="l")
lines(x=1:length(panel$dat3), y=panel$sca*panel$dat4, col="red")
grid()
panel
}
#Build the panel
xlimdat <- c(1, length(x1))
ylimdat <- c(-5, 5)
panel <- rp.control(title = "Eye-Ball-It", dat1=ylimdat, dat2=xlimdat, dat3=x1, dat4=x2, off=100, sca=1.0, size=c(300, 160))
rp.slider(panel, var=off, from=-500, to=500, action=lvm.draw, title="Offset", pos=c(5, 5, 290, 70), showvalue=TRUE)
rp.slider(panel, var=sca, from=0, to=2, action=lvm.draw, title="Scale", pos=c(5, 70, 290, 90), showvalue=TRUE)

It sounds like you want to minimize the function (Ax'+By) + (Az'+Bx) + (Ay'+Bz) for a pair of values: Namely, the time-offset: t0 and a time scale factor: tr. where Ax' = tr*(Ax + t0), etc..
I would look into SciPy's bivariate optimize functions. And I would use a mask or temporarily zero the data (both Ax' and By for example) over the "gaps" (assuming the gaps can be programmatically determined).
To make the process more efficient, start with a coarse sampling of A and B, but set the precision in fmin (or whatever optimizer you've selected) that is commensurate with your sampling. Then proceed with progressively finer-sampled windows of the full dataset until your windows are narrow and are not down-sampled.
Edit - matching axes
Regarding the issue of trying to identify which axis is co-linear with a given axis, and not knowing at thing about the characteristics of your data, i can point towards a similar question. Look into pHash or any of the other methods outlined in this post to help identify similar waveforms.