I have a program that sniffs network data and stores it in a database using pcapy (based on this). I need to make the data available in realtime over a network connection.
Right now when i run the program it will start a second thread for the sniffer and a Twisted server on the main thread, however i have no idea how to get clients to 'tap into' the sniffer that's running in the background.
The end result should be that a client enters an url and the connection will be kept open until the client disconnects (even when there's nothing to send), whenever the server has network activity the sniffer will sniff it and send it to the clients.
I'm a beginner with Python so i'm quite overwhelmed so if anyone could point me in the right direction it would be greatly appreciated.
Without more information (a simple code sample that doesn't work as you expect, perhaps) it's tough to give a thorough answer.
However, here are two pointers which may help you:
Twisted Pair, a (unfortunately very rudimentary and poorly documented) low-level/raw sockets networking library within Twisted itself, which may be able to implement the packet capture directly in a Twisted-friendly way, or
The recently-released Crochet, which will allow you to manage the background Twisted thread and its interactions with your pcapy-based capture code.
Related
I want to be able to schedule delivery of a lightweight message from a server to a client. This is new territory to me so I'd appreciate some advice on the possible approaches available.
The client is running on a Raspberry Pi using node.js (because I'm using node libraries to control a piece of attached hardware). Eventually there will be multiple clients like it.
The server could be anything, though I'm most familiar with Python, django and node.
I want to be able to access the server from a browser and cause it to schedule a future message to the client, effectively a push notification with a tiny bit of data.
I'm looking at pub-sub and messaging systems to do this; I started writing a system that uses node on both ends and sockets, but the approach I want is more fire-and-forget occasional messages, not constant realtime data exchange. I'm also not a huge fan of the node-cron style scheduling, I'd like to be able to retrieve and alter scheduled events and it felt somewhat heavy-handed to layer this on top of a cron system.
My current solution uses python on the server (so I can write a django web interface) with celery and rabbitmq, using a named queue per client. The client subscribes to that specific queue using node-amqp, and off we go. This also allows me to create queues that multiple clients can be interested in, which is a neat bonus.
This answer makes me think I'm doing the right thing -- but as I'm new to this stuff, it feels like I might be missing something. Are there alternatives I should consider in the world of server-client messaging?
Since you are already using python you could take a look at python remote objects, (pyro).
The end result I am trying to achieve is allow a server to assign specific tasks to a client when it makes it's connection. A simplified version would be like this
Client connects to Server
Server tells Client to run some network task
Client receives task and fires up another process to complete task
Client tells Server it has started
Server tells Client it has another task to do (and so on...)
A couple of notes
There would be a cap on how many tasks a client can do
The client would need to be able to monitor the task/process (running? died?)
It would be nice if the client could receive data back from the process to send to the server if needed
At first, I was going to try threading, but I have heard python doesn't do threading correctly (is that right/wrong?)
Then it was thought to fire of a system call from python and record the PID. Then send certain signals to it for status, stop, (SIGUSR1, SIGUSR2, SIGINT). But not sure if that will work, because I don't know if I can capture data from another process. If you can, I don't have a clue how that would be accomplished. (stdout or a socket file?)
What would you guys suggest as far as the best way to handle this?
Use spawnProcess to spawn a subprocess. If you're using Twisted already, then this should integrate pretty seamlessly into your existing protocol logic.
Use Celery, a Python distributed task queue. It probably does everything you want or can be made to do everything you want, and it will also handle a ton of edge cases you might not have considered yet (what happens to existing jobs if the server crashes, etc.)
You can communicate with Celery from your other software using a messaging queue like RabbitMQ; see the Celery tutorials for details on this.
It will probably be most convenient to use a database such as MySQL or PostgreSQL to store information about tasks and their results, but you may be able to engineer a solution that doesn't use a database if you prefer.
So i been readying for a awhile now. And it seems like asynchronous socket handling would be a better approach to dealing with what I'm trying to do.
Right now I'm working on a gaming server. At the moment socket server will do ok with about 3 clients or so. Sending data at the same exact time.
But my problem is, after that things start to get laggy. So if i do a asynchronous server in the same manner to what i'm already doing. Would it make the game data transfer more smoothly?
This is in python by the way.
Asynchronous sockets are more effective then synchronous. But if the game is lagging for 4+ clients, then your server/client system is badly written and it is not the matter of sockets imho.
I want to write a Python script that will check the users local network for other instances of the script currently running.
For the purposes of this question, let's say that I'm writing an application that runs solely via the command line, and will just update the screen when another instance of the application is "found" on the local network. Sample output below:
$ python question.py
Thanks for running ThisApp! You are 192.168.1.101.
Found 192.168.1.102 running this application.
Found 192.168.1.104 running this application.
What libraries/projects exist to help facilitate something like this?
One of the ways to do this would be the Application under question is broadcasting UDP packets and your application is receiving that from different nodes and then displaying it. Twisted Networking Framework provides facilities for doing such a job. The documentation provides some simple examples too.
Well, you could write something using the socket module. You would have to have two programs though, a server on the users local computer, and then a client program that would interface with the server. The server would also use the select module to listen for multiple connections. You would then have a client program that sends something to the server when it is run, or whenever you want it to. The server could then print out which connections it is maintaining, including the details such as IP address.
This is documented extremely well at this link, more so than you need but it will explain it to you as it did to me. http://ilab.cs.byu.edu/python/
You can try broadcast UDP, I found some example here: http://vizible.wordpress.com/2009/01/31/python-broadcast-udp/
You can have a server-based solution: a central server where clients register themselves, and query for other clients being registered. A server framework like Twisted can help here.
In a peer-to-peer setting, push technologies like UDP broadcasts can be used, where each client is putting out a heartbeat packet ever so often on the network, for others to receive. Basic modules like socket would help with that.
Alternatively, you could go for a pull approach, where the interesting peer would need to discover the others actively. This is probably the least straight-forward. For one, you need to scan the network, i.e. find out which IPs belong to the local network and go through them. Then you would need to contact each IP in turn. If your program opens a TCP port, you could try to connect to this and find out your program is running there. If you want your program to be completely ignorant of these queries, you might need to open an ssh connection to the remote IP and scan the process list for your program. All this might involve various modules and libraries. One you might want to look at is execnet.
I'm looking for some general information on how I should approach a problem that I think Twisted is a great fit for. (I'm new to Twisted but not Python)
I have a home automation controller that can support a single TCP socket connection, sending and receiving binary data. I'd like to use XMPP as a bridge to the socket so a user can send commands and receive events.
I got a rudimentary socket connection working with Twisted that was able to send and receive commands from one of the examples in the O'Reilly book. I also have a fully working Python XMPP bot written with the SleekXMPP library that I'm happy with. I'm just not sure how to bring these together.
The basic scenario is:
User sends message to XMPP bot, which figures out what command to send to the socket
ASCII Socket command is converted to binary and sent to socket
Socket receives command and sends binary response
Binary response converted to ASCII
XMPP bot sends response back to user.
Network events (independent from user action) can also be received by network socket and should be sent to user
It's #6 that is presenting the challenge, otherwise I'd just open/close the socket on demand when in need to write something.
The part that I'm having trouble wrapping my head around with Twisted is the best approach to make these two event loops communicate. I've seen lots of info on using Queues, deferred, threads, select, etc. I have a feeling that Twisted can handle much of the complexity if I just learn to use the tool properly.
If someone can point me in the right direction, I'll take the ball and run with it. As I mentioned, I'm happy with my XMPP bot and I'd like to use the existing code. I think my problem now comes down to creating the socket in the background, then sending and receiving data from that socket in the foreground.
By the way, I'm very happy to share back my code once it's working so someone else can benefit from the help I'm asking for.
-- Scott
One of the problems with a non-blocking IO engine is that its pretty much all-or-nothing. As soon as you introduce blocking code, you can quickly lose most of the benefits of the event-driven asynch approach. Wherever possible (as a rule of thumb), its best to have the entire app running off the same reactor.
As i see it, you have two options:
Twisted is not thread safe. That said, you can use mechanisms like deferToThread and callFromThread to interact with other threads. This is by far the most confusing and needlessly complex approach for your application design. It's particularly painful if you're new to twisted.
Use twisted.words.protocols.jabber, and implement your XMPP stuff in a non-blocking manner using the twisted reactor. That way it will happily exist alongside all your other twisted code. and allow you to cleanly interact between protocols. It will result in less code, and a robust implementation that is easy to extend, maintain, and test.