I'm trying to use Python to 'mic-monitor', i.e., to playback the mic signal through the headphones in real-time, in addition to any other output signal from the PC.
I can accomplish this by amending my PC's playback settings, but I want to do it with Python, so that I can program a Raspberry Pi to mic-monitor my cheap headset for the PS4.
I can almost accomplish this with PyAudio or sounddevice, but there is a small but significant delay. So:
Is there a way to eradicate that latency with Python, for example by somehow accessing my PC's inputs more directly?
Otherwise, why is it not possible with Python?
Can I somehow configure the Raspberry Pi to mic monitor in the same way as my PC?
Sounddevice code is shown below for reference:
import sounddevice as sd
duration = 5.5 # seconds
def callback(indata, outdata, frames, time, status):
if status:
print(status)
outdata[:] = indata
with sd.Stream(channels=2, callback=callback):
sd.sleep(int(duration * 1000))
There will always be latency with a computer in-between. Professional audio gear is usually custom built for minimal latency (or it's just analog). To reduce latency you need to record smaller chunks at a time before sending them to the output which does introduce more processing overhead. Using smaller chunks can also at some point introduce more jitter in the signal because the inter-frame latency might not keep up with the sample rate. PortAudio is probably likely able to be configured to have a bit less latency, but you're probably getting a lot of the delay from the OS and audio drivers as well. Here's a page discussing how you can optimize the OS and audio drivers for minimal latency on a Raspberry Pi. PortAudio (the driver behind most python audio libraries), also has a discussion on audio driver latency based on your operating system.
looking at the documentation for sd.Stream, it looks like even if you specify a smaller blocksize, due to the implementation, it may make latency even worse.
There is however an option to specify an exact latency (if a particular delay is desirable) or to achieve a best effort "as fast as possible" by specifying latency = "low" This attempts to take into account the specific hardware you're using, and go as fast as possible.
Related
I am developing a python application where a drone and a computer communicate over local network (wifi). My need is to stream the drone's camera to OpenCV-python on the computer with the lowest possible latency at the highest possible resolution.
Thus far I have been trying rather naive approaches over TCP that give okay-ish results, I get something like 0.1s or 0.2s latency for VGA format. It has a point for some use cases as it enables lossless transmission, but since the most common scenario is to aggressively control the drone in real time from the stream, I am aiming for something of much lower latency and hopefully higher resolution.
My advisor has recommended using WebRTC. I have done some research on the matter, found the aiortc library that implements WebRTC in python, but I am unsure this is the way to go for my use case as it seems to be more geared toward web developers.
I am a bit lost I think. Could you highlight the advantages of WebRTC in my application if any, or point me toward solutions that are more relevant for my problem please?
Thanks in advance!
[1]
rtc communicate peer to peer, I think you knew that. And if u use local network U will not need STUN server or TURN to connect two devices. That is make more decrease latency and code shorter. I'm not work with drone but I think your method stream had latency < 0,2 is good.
fyi protocol campare
I'm developing a program in Python that uses UDP to receive data from an FPGA (a data collector device). The speed is very high, about 54 MB/s at the highest setting, that's why we use a dedicated gigabit ethernet connection. My problem is: a lot of packages get lost. This is not a momentary problem, the packets come in for a long time, then there's a few seconds long pause, then everything seems fine again. The pause depends on the speed (faster communication, more lost).
I've tried setting buffers higher, but something seems to be missing. I've set self.sock_data.setsockopt(socket.SOL_SOCKET,socket.SO_RCVBUF,2**28) to increase buffer size along with the matching kernel option: sysctl -w net.core.rmem_max=268435456.
Packages have an internal counter, so I know which one got lost (also, I use this to fix their order). An example: 11s of data lost, around 357168 packages. (I've checked, and it's not a multiple of an internal buffer size in either of my program or the FPGA's firmware). I'm watching the socket on a separate thread, and immediately put them into a Queue to save everything.
What else should I set or check?
I'm working on a project where one client needs to take several snapshots from a camera (i.e. it's actually taking a short-duration video, hence a stream of frames), then send all images to a server which then performs some processing on these images and returns a result to the client.
Client and server are all running Python3 code.
The critical part is the image sending one.
Some background first, images are *640*480 jpeg* files. JPEG was chosen as a default choice, but lower quality encoding can be selected as well. They are captured in sequence by a camera. We have thus approximately ~600 frames to send. Frame size is around 110KiB.
The client consists of a Raspberry Pi 3 model B+. It sends the frames via wifi to a 5c server. Server and client both reside in the same LAN for the prototype version. But future deployments might be different, both in term of connectivity medium (wired or wireless) and area (LAN or metro).
I've implemented several solutions for this:
Using Python sockets on the server and the client: I'm either sending one frame directly after one frame capture, or I'm sending all images in sequence after the whole stream capture is done.
Using Gstreamer: I'm launching a GStreamer endpoint on my client and directly send the frames to the server as I stream. I'm capturing the stream on the server side with OpenCV compiled with GStreamer support, then save them to the disk.
Now, the issue I'm facing is that even if both solutions work 'well' (they get the 'final' job done, which is to send data to a server and receive a result based on some remote processing), I'm convinced there is a better way to send a large amount of data to a server, using either the Python socket library, or any other available tools.
All personal researches are done on that matter lead me to solutions similar to mine, using Python sockets, or were out of context (relying on other backends than pure Python).
By a better way, I assume:
A solution that saves bandwidth as much as possible.
A solution that sends all data as fast as possible.
For 1. I slightly modified my first solution to archive and compress all captured frames in a .tgz file that I send over to the server. It indeed decreases the bandwidth usage but also increases the time spent on both ends (due to the un/compression processes). It's obviously particularly true when the dataset is large.
For 2. GStreamer allowed me to have a negligible delay between the capture and the reception on my server. I have however no compression at all and for the reasons stated above, I cannot really use this library for further development.
How can I send a large number of images from one client to one server with minimal bandwidth usage and delay in Python?
If you want to transfer images as frames you can use some existing apps like MJPEG-Streamer which encode images from a webcam interface to JPG which reduces the image size. But if you need a more robust transfer with advanced encoding you can use some Linux tools like FFMPEG with streaming which is documented in here.
If you want lower implementation and control the whole stream by your code for modifications you can use web-based frameworks like Flask and transfer your images directly throw HTTP protocol. You can find a good example in here.
If you don't want to stream you can convert a whole set of images to a video encoded format like h264 and then transfer bytes throw the network. You can use opencv to do this.
There are also some good libraries written in python like pyffmpeg.
you can restream camera using ffmpeg over network so that client can read it both ways. it will reduce delays.
I'm having trouble carrying out what I think should be a pretty straightforward task on a NIDAQ usb6002: I have a low frequency sine wave that I'm measuring at an analog input channel, and when it crosses zero I would like to light an LED for 1 second. I'm trying to use the nidaqmx Python API, but haven't been able to clear up some of my basic questions with the documentation. https://nidaqmx-python.readthedocs.io/en/latest/
If anyone can offer any thoughts about the code or the basic logic of my setup, that would be hugely helpful.
Here's what I have tried so far. I start with some imports and the definition of my channels:
import matplotlib.pyplot as plt
from math import *
import nidaqmx
from nidaqmx import *
from nidaqmx.constants import *
import time
V_PIN = "Dev1/ai6"
LED_PIN = "Dev1/ao0"
I understand how tasks and things work generally- I can read and plot a signal of a given sampling rate and number of samples using task.ai_channels methods without any trouble. But here's my best guess at how to carry out "detect zero and trigger output":
writeLED = nidaqmx.Task('LED')
writeLED.ao_channels.add_ao_voltage_chan(LED_PIN)
writeLED.timing.cfg_samp_clk_timing(1)
writeLED.triggers.start_trigger.cfg_anlg_edge_start_trig(V_PIN,trigger_level = 0)
writeLED.write([5], auto_start=True)
This gives me the error below at the cfg_anlg_edge line
DaqError: Requested value is not a supported value for this property. The property value may be invalid because it conflicts with another property.
Property: DAQmx_StartTrig_Type
Requested Value: DAQmx_Val_AnlgEdge
Possible Values: DAQmx_Val_DigEdge, DAQmx_Val_None
I don't know why an analog input channel wouldn't be supported here. Page 245 of this document makes it sound like it should be: https://media.readthedocs.org/pdf/nidaqmx-python/latest/nidaqmx-python.pdf
I'm sure there are other problems with the code, too. For example, it seems like the sample clock manipulations are quite a bit more complicated than what I've written above, but I haven't been able to find anything that explains how it would work in this situation.
Thanks in advance for any help!
With NI, it's "RTFMs"
When programming NI devices, you usually need two manuals.
NI-DAQmx Help (for the programming part)
the device specification (for the device part)
You need both because the NI-DAQmx API supports every DAQ device NI makes, but not every device has the same capabilities. "Capabilities" includes more than how many channels of each kind, but also the timing and triggering subsystems as well as internal signal routing. A DAQmx application that runs with one device is not guaranteed to run with another because the application might use the API in a way the second device cannot support.
Finally, on the documentation front, any given NI DAQ device typically belongs to family of related devices and these families also have a manual called User Guide. These User Guides act as a bridge between the API and device spec, helping you understand how the device responds to commands. For the 6002, the family is "Low-Cost DAQ USB Device".
Analog trigger for analog output on NI 6002
Your determination is correct that
writeLED.triggers.start_trigger.cfg_anlg_edge_start_trig(V_PIN,trigger_level = 0)
is possible, just not for the USB 6002. This line is asking the analog output subsystem to use an analog edge trigger, but the analog output subsystem for the 6002 only has these trigger capabilities:
software
PFI 0
PFI 1
For this device, you're only option is the software trigger because the PFI lines are digital triggers and their trigger level is specified to be between 0.8 V and 2.3 V.
Change your Python program to detect a zero-crossing from the analog input stream and, when it does, make it call stop() and then start() on the AO task.
The reason for the stop-start sequence is retriggering: you want to light the LED for each zero crossing, but a task cannot be restarted unless it has either been stopped (by the API or by completing its task) or configured for retriggering. Because the 6002 is in the low-cost family, this hardware feature isn't available, so you must use the API to stop the AO task or wait for the AO generation to complete before restarting the pulse for the LED
6002 AO Specification
Software triggering is not real-time, you will have non-deterministic delay before the led turns on. This depends on your program, interfaces, usb latencies, pc performances...
Otherwise, you can use a comparator (like lm393) to trigger a digital input (PFI0 or PFI1).
Though it's just an LED, it is probably not critical if the delay varies within milliseconds.
I am looking for a simple means of triggering my data acquisition software using an external TTL pulse. I need to sample data from multiple sources synchronously with a 5 Hz reference clock. The acquisition does not need real-time priority but I want to ensure that my software is triggered as soon as possible and exactly once per external clock cycle. I would prefer to do this by somehow getting an interrupt from the external trigger without needing to use a fast polling loop. As far as I can tell, you can't just use a parallel port pin for an interrupt in a modern OS like Linux. Any ideas?
I am also thinking of generating broadcast packets on my network to notify other machines on the network that a trigger event has occurred. Due to network latency however there may not be enough time available in the 200ms period between triggers to do the acquisition.
Rather than use the parallel port, have you considered using a serial device? Since you have a TTL signal, you'll possibly need a level converter to convert TTL to RS232 +/- 12V levels. Once you're using a serial device, you can use the standard serial ioctl() calls to detect a change in control signal status.
Specifically, you could use the TIOCMIWAIT ioctl on the connected serial device to wait for a change on say the DCD line, which you would connect to your clock source.
Your user space application would be blocked waiting in the TIOCMIWAIT ioctl system call until there is a change of status on your clock line, at which point your app would become runnable and return from the ioctl. You might have to take care to ensure that you handle the case where you get a change of status interrupt on both rising and falling edges of your serial control signals. On some UART hardware (eg TL16C554A UART) it's possible that you'll only get an interrupt for a signal transitioning in a single direction. For the TL16C554A for example, the TIOCMIWAIT would only fall through on the rising edge of any Ring Indicate signal change.
Using the serial ioctls in this manner also has the advantage that you could use a USB-Serial dongle that supports TIOCMIWAIT if required (eg PL2303), and still retain user level software compatibility, albeit at the expense of increased latency due to USB.
If you require lower latency than can be achieved through user space, you'd be best to write a kernel driver module which can handle the timing and sampling, but I wouldn't suggest this route unless absolutely needed. It's easier to develop user space code.
Here's some incomplete sample C code snippets for using the TIOCMIWAIT ioctl.
int serial_fd = open(cmdline.device_name, O_RDWR | O_NONBLOCK | O_NOCTTY);
static const unsigned int ri_flag = TIOCM_RNG;
/* Set up serial port here using tcsetattr. Set CRTSCTS | CLOCAL to ensure status interrupts
* are generated.
*/
while (1) {
/* Wait for positive RI transition. TIOCMIWAIT takes a mask
* as argument, only returning when the appropriate signal has changed.
*/
if (ioctl(serial_fd, TIOCMIWAIT, ri_flag)) {
fprintf(stderr, "ioctl() failed waiting for RI edge [%s]\n", strerror(errno));
break;
}
/* Do sensor sampling here. You could use TIOCMGET to first verify that
* the clock line is in the expected state, eg high, before continuing.
*/
}
Polling is a fine method for such a slow data rate. Poll at 1 ms. That should be fine. Trying to use a hardware interrupt is going to cause much pain.
Google for "Interrupt Linux GPIO" if you want to do it the hard way. :)
https://developer.ridgerun.com/wiki/index.php/How_to_use_GPIO_signals
Consider connecting the external pulse source to the 'CD' ping of a real (! not a USB to RS232 converter) serial port. Then you can use the "PPS api" to get an as exact timestamp from which the pin "went high" as possible.
You might need a TTL signal shifter; not all serial ports work correctly with TTL signal levels.
The PPS api is normally used for time keeping. E.g. connect the PPS pin of a GPS module to your pc and let NTP sync with that. This gives you microsecond accuracy.
This PPS api is supposed to be more accurate than any other userspace solution (e.g. the TIOCMIWAIT ioctl) as it is completely handled in the kernel, immediately when the interrupt (as triggered by the CD signal change) comes in. With the ioctl solution you have at least a context switch. I did some testing on a raspberry pi and a userspace solution gives at least 6us jitter.
The PPS api gives you a timestamp from when the pulse was detected.
Why not use a USB to RS232 converter: I read somewhere (in relation to timekeeping) that USB devices are polled once every +/- 1ms. This polling frequency also depends on how busy the system is with other USB devices and I think the internal clock of the USB device may also influence things. I've never measured this though.
Relevant URLS:
https://www.rfc-editor.org/rfc/rfc2783 RFC describing the PPS API
http://linuxpps.org/ Linux implementation the PPS API (it is in the standard kernel)
http://comments.gmane.org/gmane.comp.hardware.gps.gpsd.user/3747 a thread which mentions why not to use USB for timekeeping
Regarding the network functionality: use UDP broadcasts, do not use TCP.
I ended up using the serial port CTS line for the trigger using the TIOCMIWAIT ioctl per Austin Phillips answer. Since RS232 requires +/-12V levels I was able to get the necessary power for this level shifter from the other serial control lines.
The Python code to implement this solution can be found in question: Python monitor serial port (RS-232) handshake signals