So I have tried to find a answer but must not be searching correctly or what I'm trying to do is the wrong way to go about it.
So I have a simple python script that creates a chess board and pieces in a command line environment. You can in put commands to move the pieces. So one of my co workers thought it would be cool to play each other over the network. I agreed and tried by creating a text file to read and write to on the network share. Then we would both run the script that reads that file. The problem I ran into is I pretty much DOS attacked that file share since it kept trying to check that file on network share for a update.
I am still new to python and haven't ever wrote code that travels the internet, our even a simple local network. So my question is how should I go about properly allowing 2 people to access this data at the same time with out stealing all the network resources?
Oh also im using version 2.6 because thats what everyone else uses and they refuse to change to new syntax
You need to use the proper networking way. It's not quite hard for simple networked program like yours.
Use the one from the Python's stdlib http://docs.python.org/library/socket.html (also take a look at the examples at the bottom of the page).
First off, without knowing how many times you are checking the fle with the moves, it is difficult to know why the file-share is getting DoS-ed. Most networks and network shares these days can handle that level of traffic - they are all gigabit Ethernet, so unless you are transferring large chunks of data each time, you should be ok. If you are transferring the whole file each time, then I'd suggest that you look at optimizing that.
That said, coming to your second question on how this is handled at a network level, to be honest, you are already doing it in a certain way - you are accessing a file on a network share and modifying it. The only optimization required is to be able to do it efficiently. Even over the network operations in a concurrent world do the same. In that case, it will be using fast in-memory database storing various changes / using a high-scale RDBMS / in the case of fast-serving web-servers better async I/O.
In the current case, since there are two users playing the game, I suggest that you work on a way to transmit only the difference in the moves each time over the network. So, instead of modifying the file over the network share, you can send the moves over to a server component and it synchronizing the changes locally to the file. Of course, this means you will need to create a server component that would do something like this
user1's moves <--> server <--> user2's moves . Server will modify the moves file.
Once you start doing this, you get into the realm of server programming / preventing race conditions etc. It will be a good learning experience.
Related
Before I buy my first setup. I'll launch my deep-learning-pipline on sth like vast.ai.
I never did it before, but How can I protect my script from being "stolen"?
This should be a serious launch and take around 7 days to finish training.
google colab doesn allow enough memory & ram for what i need ( need around 64GB ram)
is there a way to run a python script encrypted? (note: it makes use of libaries)
It is hard to run python encrypted. However, you could try to store the code into encrypted disk space.
There are some ways, from fully obfuscating your script to converting your script into equivalent cython or creating an executable out of it using the likes of Nuitka.
You may also implement some important logic in C/C++ (as extensions) and then call them in your script.
You may also set up a server where you feel is OK and send the bits that needs to be executed, basically create a distributed system.
As you can see there are many ways, and the deeper you go the more complex it gets.
Also you might want to have a look here as well.
I have to setup a program which reads in some parameters from a widget/gui, calculates some stuff based on database values and the input, and finally sends some ascii files via ftp to remote servers.
In general, I would suggest a python program to do the tasks. Write a Qt widget as a gui (interactively changing views, putting numbers into tables, setting up check boxes, switching between various layers - never done something as complex in python, but some experience in IDL with event handling etc), set up data classes that have unctions, both to create the ascii files with the given convention, and to send the files via ftp to some remote server.
However, since my company is a bunch of Windows users, each sitting at their personal desktop, installing python and all necessary libraries on each individual machine would be a pain in the ass.
In addition, in a future version the program is supposed to become smart and do some optimization 24/7. Therefore, it makes sense to put it to a server. As I personally rather use Linux, the server is already set up using Ubuntu server.
The idea is now to run my application on the server. But how can the users access and control the program?
The easiest way for everybody to access something like a common control panel would be a browser I guess. I have to make sure only one person at a time is sending signals to the same units at a time, but that should be doable via flags in the database.
After some google-ing, next to QtWebKit, django seems to the first choice for such a task. But...
Can I run a full fledged python program underneath my web application? Is django the right tool to do so?
As mentioned previously, in the (intermediate) future ( ~1 year), we might have to implement some computational expensive tasks. Is it then also possible to utilize C as it is within normal python?
Another question I have is on the development. In order to become productive, we have to advance in small steps. Can I first create regular python classes, which later on can be imported to my web application? (Same question applies for widgets / QT?)
Finally: Is there a better way to go? Any standards, any references?
Django is a good candidate for the website, however:
It is not a good idea to run heavy functionality from a website. it should happen in a separate process.
All functions should be asynchronous, I.E. You should never wait for something to complete.
I would personally recommend writing a separate process with a message queue and the website would only ask that process for statuses and always display a result immediatly to the user
You can use ajax so that the browser will always have the latest result.
ZeroMQ or Celery are useful for implementing the functionality.
You can implement functionality in C pretty easily. I recomment however that you write that functionality as pure c with a SWIG wrapper rather that writing it as an extension module for python. That way the functionality will be portable and not dependent on the python website.
We have begun upgrading hardware and software to a 64-bit architecture using Apache with mod_jk and four Tomcat servers (the new hardware). We need to be able to test this equipment with a large number of simultaneous connections while still actually doing things in the app (logging in, etc.)
I currently am using Python with the Mechanize library to do this, but it's just not cutting it. Threading is not "real" in Python, and multiprocessing makes the local box work harder than the machines we are trying to test since it has to load so much into memory for Mechanize.
The bottom line is that I need something that will really hammer this thing's connections and hold a session to make sure that the sticky sessions are working in mod_jk. I need to be able to code it quickly, it needs to be lightweight, and being able to do true multithreading would be a perk. Other than that, I am open-minded.
Any input will be greatly appreciated. Thanks.
Open Source Testing Tools
Not knowing the full requirements makes it difficult, however something from the list might fit the bill.
In order to accomplish what I wanted to do, I just went back to basics. Mechanize is somewhat bulky, and there was a lot of bloat involved in the main functionality tests I had before. So I started with a clean slate and just used cookielib.CookieJar and urllib2 to build a linear test and then run them in a while 1 loop. This provided enough strain on the Apache system to see how it would react in the new environment, and for the record, it did VERY well.
We have a process here at work where an input file is created by SAS. That input file is then read by a legacy application and that legacy application creates results. SAS then reads the results and summarizes them. A non-programmer usually takes care of these actions one by one. So the person just creates the input file. They know when it is done, and then they run the legacy application, and they know when that is done. Then they run the summary program.
I have a situation where my boss wants to run about 100 variations. I have access to 3 or 4 computers that share a network drive. Here is my plan: Use computer A, I start creating the 100 input files, one by one. Using computer B, I run the legacy program on each input file. I would like to start running the program when input is ready. So if input1 is done being created on computer A, I want to run the legacy application on input1 on computer B while input2 is being created on computer A. I know python best, so I will probably use python to glue all of this together.
Now I know there is a lot of things I could do, but I think this approach is adequate and will allow me to just get the job done for the time being. I don't really have time to design and test a very elegant solution that would take advantage of all the cores on all the machines or use a database to help me synchronize all of this. I appreciate suggestions like this, but I really just want to know if there is a way, in python, to tell if a file on a network drive is open for writing by any application on any computer? If not, I will probably just come up with a dumb way to create an indicator that the job is done - like create a file "doneA" where if it exists, it means the "input1" file is complete. For example. I would add a step to the sas program that creates an indicator file after the input file has been created.
Sorry for the really long explanation, but I just don't want you to waste your time offering alternate solutions that I probably will not be able to implement.
I have read this question and its responses. I don't think I can use anything like lsof b/c these files would be open on different computers.
Write output to a temp file. When done writing, close it, then rename it to the name the other program is waiting for. That way, the file only appears when it is ready to be read.
if there is a way, in python, to tell if a file on a network drive is open for writing by any application on any computer?
Not really.
Windows will let you open the file several times and really muck things up.
You have to use some explicit synchronization. Rather than synchronize each of the three steps 100 different ways, my preference is to do the following. Create 100 copies of the three-step dance. Don't worry about synchronization among the steps.
for variant in range(100):
name= "variant_{0}.bat".format(variant)
with open(name,"w") as script:
print( "run some SAS thing", file=script )
print( "run some legacy thing", file=script )
print( "run some SAS thing", file=script )
subprocess.Popen( "start {0}".format(name), shell=True )
I suspect that this will crush the life out of your processor by running all 100 in parallel.
As a practical matter, you probably don't want to actually use subprocess.Popen() in Python. As a practical matter you probably want to create several of these "start variant_x" batch files that can run a few variants in parallel. You can create some kind of master bat file that runs a sequence of processing steps. Each step starts several parallel 3-step variants.
When an audio or midi clip is played (triggered), its name needs to be sent using OSC to another application.
LiveAPI is an interface which allows one to explore and automate Ableton Live using python scripts.
The code to do this must be written in a python script, which must be placed in a specific folder where Ableton Live can find it, selected in Live's Preferences.
More information about the LiveAPI can be found on these sites:
http://www.assembla.com/wiki/show/live-api
http://groups.google.com/group/liveapi
According to the LiveAPI documentation, the Clip object has a "name" attribute which holds the clip name. Presumably that's what you want to send in your OSC packets.
Also, it's worth mentioning that the Max/MSP support in Live8 will probably be a lot more comfortable to work with than LiveAPI, which is pretty much a dead project. Max/MSP supposedly has OSC support, which was added to support the JazzMutant Lemur, but I'm not sure how much of that made it into Live. Anyways, it's worth keeping in mind for when Live8 is released.
I know about Max 4 Live, but as I see it, it's kind of a different thing. Yes, it will probably be able to interface with Live to do all the stuff which people do now with LiveAPI. Some even think that M4L may not even go through LiveAPI, and use some internal interface instead (since Ableton and Cycling 74 are developing it together). From the promo videos on ableton.com site I think that M4L will mostly be about making and modifying sound, and not so much about controlling/reading other instruments, effects, clips etc.
I would not say that LiveAPI project is dead, because a lot of hardware MIDI controllers rely on LiveAPI to do some auto-mapping magic. When you look at the MIDI Remote Scripts folder in Live, you'll see that each controller has it's own folder with a python script. So I definitely think that LiveAPI is going to stay, and that this door into Live will remain open. They even created a new folder called Framework which contains some newer code, probably required for the new Akai controller to work with Live (that is what people believe in theory).
The application I plan to use the playing clip's name is called vvvv, so I don't want to have to bring Max into this, because it is not really needed.
I had some success with someone's modification of the original LiveAPI code, but only worked when I request all the clips' names, not when I asked for just a single one. I didn't have time to play with it later, and the thing for which I was preparing this has passed. I plan to work that out eventually, but it's not that urgent anymore.