Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
I would like to ask a question about transferring data from c++ to python in real time.
My situation is :
1) I am generating data every 1 ms in c++,
2) I would like to stack data for certain amount of time and make a dataset,
3) I would like to run some machine learning algorithm written in Python without turning of c++ program.
So far, I have thought about several things :
option 1 ) Save the dataset as a txt file and read it from python. But this seems too slow due to I/O process.
option 2 ) Use IPC such as zeromq. I am quite new to IPC, so I am not sure if it is the thing I really want. Also, among the multiple methods (mmap, shared memory, message queue, ...), I do not know which one is the best shot for me.
option 3 ) Use UDP. From my understanding UDP sometimes sends the same data twice, or skips the data, or mixes data up (e.g. previous time step data arrives later)
Is there any recommendations I need to search and study?
Related
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 2 years ago.
Improve this question
We have a streaming pipeline reading from pub/sub and writing to bigquery. It wasn't working without adding a window function, because a default global window only fires once and doesn't know when to re-trigger. There is no GroupBy or combine.
We tried to add a beam Window with a trigger, but there are some problems. If we use a globalWindow, it runs really slow and sometimes gives null pointer exceptions. If we use a fixed window, it's fast but but it doesn't seem to acknowledge the pub/sub messages sometimes.
What we'd really want is a pipeline that reads from pub/sub, gets a batch of however many it could get, writes to bigquery, and once everything is written and the pubsub messages are acknowledged, retrigger the read-from-pubsub. Is this possible?
I think you are looking for this. You have a composite trigger named Repeatedly.forever and you can combine it with AfterCount
Something like this where you trigger after 1000 elements read.
Repeatedly.forever(AfterCount(1000))
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
For background, I am somewhat of a self-taught Python developer with only some formal training with a few CS courses in school.
In my job right now, I am working on a Python program that will automatically parse information from a very large text file (thousands of lines) that's a output result of a simulation software. I would like to be doing test driven development (TDD) but I am having a hard time understanding how to write proper unit tests.
My trouble is, the output of some of my functions (units) are massive data structures that are parsed versions of the text file. I could go through and create those outputs manually and then test but it would take a lot of time. The whole point of a parser is to save time and create structured outputs. Only testing I've been doing so far is trial and error manually which is also cumbersome.
So my question is, are there more intuitive ways to create tests for parsers?
Thank you in advance for any help!
Usually parsers are tested using a regression testing system. You create sample input sets and verify that the output is correct. Then you put the input and output in libraries. Each time you modify the code, you run the regression test system over the library to see if anything changes.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
I am a python beginner and I am a little experienced in OO-programming in Java and PHP and also fucntional programming in R . Thus, my question is considering the general usage of python scripts in everyday use-cases.
I want to "learn" how to think/approach a problem that I do experience when facing a situation with my software where a "script" could help me out or improve something.
For instance, I've heard friends talking about their self-made python scripts to evenly mute the audio of movies to avoid loud outliers in explosive scenes, etc. Another example, in my case righ tnow, is to filter out certain pictures with no GPS-time meta information for the timezone in order to sort these fotos in accordance with the others.
I really want to get the essence and recipe based on the aforementioned examples to better integrate Python in my everyday life and get an intuitive feeling for it. (i.e. how would a simple script look like that takes a picture, filters out its meta data, and does something -> where do I have to run the script so I can call the function with these .JPG files as its arguments?).
I would also be glad if some of you could recommend some practical tutorials or literature.
Thank you in advance :)
P.S. I know it is not a concrete question but rather it is intended to get a glimpse on a wide field of usage and thinking - but I want to get this essential take away that motivates me and shows me the direction.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
I have spent the last six months learning python as a way to automate my working environment. So far I have automated data extraction and report downloading from various web-based sources through the use of webcrawlers, interacted with excel files, created visual representations of data through matplotlib, and removed almost all the monotony from bank reconciliation.
I now come to a new task which takes up a large amount of my daily workload. We use an accounts program called Sage 50 Accounts. I effectively want to begin to learn how to manipulate the data contained within this program so that my daily thought patterns can be put into Python code.
Because this hasn't been done, there's no pre-made API. So my question is:
When wishing to interact with a new program through Python, how does a programmer begin such an inquiry?
Please accept that this question is only vague and general because I'm incredibly new to such a task.
SData is Sage's general data access API layer and should suit your purposes.
Otherwise you might need to invest in or obtain a Sage Development SDK.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
I needed to perform a heavy operation while using a Tkinter GUI. So the GUI would stop responding as soon as the operation began. So, I had two choices(or that's what i think,as I'm new to python & programming as well): MultiThreading or Schtasks .
So, I chose the easier of the two,i.e Schtasks, as I'm working on a deadline(& I dont know much about Multithreading).
What I'm doing is accessing a python file from a different project.
I run batch files which is in this different project(which contains the desired python file that i need to run) to be run by Schtasks
Now the constraint is batch file can access only this python file & not a particular method present in that file(isn't it?) & I need to access only a particular method .
So, my question is:
Is the approach I'm using correct? If not what do you suggest would be better ? Or should I just switch to MultiThreading
Your question opens a huge topic - what you are trying to do is generally not simple and can have large problems which you cannot even foresee if you don't know the topic of multitasking very well. One issue, for example, is synchronizing the access to the file you mention from within different threads or processes or tasks.
However, if you want to start somewhere and just want to write something which separates your GUI code from your computation code, I recommend you start here: http://docs.python.org/2/library/multiprocessing.html .