Develop Reference

Use python to naively find an Arduino's IP address from another PC on the local network

I have a ESP8266 Nodemcu device running a local HTTP server. I followed the quick-start instructions here.
My goal is to have a large number of these devices running in sync. To do that, I wrote this script:
#!/usr/bin/env python
import time
import sys
import socket
import requests
def myFunction():
#This is what I have right now...
ipAddresses = ["192.168.1.43", "192.168.1.44"]
#
#Instead, I need to search the local network for all the arduinos [ESP8266s] and append their address to the array.
#
for address in ipAddresses:
TurnOnLED(address)
def TurnOnLED(address):
try:
r = requests.post('http://' + address + '/LedON')
r.close()
except requests.exceptions.ConnectionError:
pass
#Main
def Main():
try:
print("Press CTRL-C to stop.")
while ON:
myFunction()
time.sleep(60)
except KeyboardInterrupt:
sys.exit(0)
if __name__ == "__main__":
Main()
This works, and allows me to control all of my devices from my desktop PC. My difficulty is with finding each IP address dynamically. I considered assigning a static IP address, but I would like them to be retail-optimized, so I cannot guarantee any particular IP address being unused out-of-the-box.
For the same reason, I want to be able to install the same arduino code on all of them. This is what I mean by naive. I want my python script to find the IP address of each device over the local network without any [unique] 'help' from the devices themselves. I want to install the same code on each device, and have my script find them all without any additional set up.
My first attempt was using the socket python module and looking for every hostname that began with "ESP-" since those are the first four characters of the factory hostname on all the boards. My query consistently turned up nothing, though.
This answer here has some good information, but my solution runs on a Macintosh, and I will not have the full host name, only "ESP-".

So, now that I know what are the constraints, and what is just what is in the tutorial, here are my 2 cents (keep in mind that I too, am just a hobbyist about everything that has "voltage". And not even a good one).
1st strategy : PC is server
So, if I assume that your devices are, for example, temperature sensors, and you want to frequently grab all the temperatures, then, one strategy could be that those devices all connect, every minute, to the server, report the temperature, and disconnect. For example, using a HTTPS request to do so.
So host the server on your PC, and have your devices behave as clients
# Example request from micropython
import urequests
rep = urequests.get(URL)
print(rep.text)
response.close()
2nd strategy : a third party is server
A variant of that strategy is to have a sort of central data repository, that acts as a server. So, I mean, your PC is not the server, nor any ESP, but another machine (which could be one ESP, a rasppi, or even a droplet or EC2 instance in the cloud, doesn't matter. As long as you can host a backend on it).
So, ESP8266 are clients of that server. And so is your PC. So that server has to be able to answer to request "set value from client", and "get value for the PC".
The drawback of those first 2 strategies, is that they are ok when the devices are the one sending data. It fits less if they are (as in your led example, but I surmise that was just an example) the ones receiving mostly the data. Because then, you would have to wait for the ESP to connect to have it get the message "you need to switch on the led".
3rd strategy : central registry
Or you can have it both way. That is keep your current architecture. When your PC wants something from the ESP8266, it has to connect to them, that are servers, and send it request.
But, in parallel to that, ESP8266 also behave as clients to register themselves in a central directory, which could be on the PC, or on a 3rd party. The objective of that central directory is not to gather the data from the ESP8266. Just to gather a uptodate list of them. Each minute they, in parallel with their server activity, the ESP8266 send a message "I am alive" to this central dircetory. And then the PC (that could be, or not, hosting that central directory) just need to get all IP associated with a not too old "I am alive" message, to get a list of IP.
4th strategy : ARP
Once your PC is on, it could scan the network, using an ARP request, with scapy. Search for "scanning local network with scapy".
For example here is a tutorial
From there, you get a list of IP. Associated with MAC address. And now, you need to know which ones are the ESP8266. You can here also apply several ideas. For example, you may use the MAC address to find guess which one are the ESP8266. Or you may simply try a dummy request on all found IP, to check which one are the ESP8266 (using a specific API of the server code of the ESP8266 you wrote)
Or you may decide to host the server on each ESP8266 on a specific port, like 8123, so that you can quickly rule out devices whose port 8123 is not listening.
5th strategy : don't reinvent the wheel
The best strategy is clearly a mix between my second and my third. Having a directory, and a 3rd party handling messages. But that is reinventing message brokers.
There is one well known middleware, fitted for ESP8266 (I mean, for IoT low profile devices), that is MQTT.
That needs some more tutorial reading and long trial and error on your behalf. You can start here for example. But that is just the 1st example I found on Google searching "MQTT ESP8266 micropython". There are zillions resources on that.
It may seem to be not the easiest way (compared to just copy and paste some code that list all the alive IP on a network). But in the long run, I you intend to have many ESP8266, so many of them that you can't afford to assign them static IP and simply list their IP, you probably really need a message broker like that, a preferably, not one that your reinvent

How To Find Out What Router/Switch ANOTHER Computer Is Connected To

This is somewhat of a repost from Is there a way to find out the BSSID a computer is connected to in a local network? but there was no answer there so I will try to quantify more specifically what I am trying to do here.
I'm working on a project for my job where we have a very large network. The goal is to be able to guesstimate on a map where a given machine is on the campus by finding out what AP the device is connected to. I have worked in the past on something similar (much smaller scale) with Scapy and Python, so I know it's possible. Unfortunately, and fortunately, ICMP responses are disabled, rendering tracert's that point only to the building's main switch, followed by an immediate hop to the device (missing hops).
Doing testing with netsh, I can find the BSSID that any given computer on the domain is connected to when I am physically using the machine. In essence, the goal then here is to find a way to get the machine to report to me what the BSSID is of its own network connection, which would allow me to reference our network map and work on that.
I am using Python for the bulk of this program, as I've worked with Scapy before on similar tasks. I have code all prepared for resolving IP from hostname, mac from IP, etc etc. However, these functions are all working so until I resolve this there is no point in including that code in this post. I'm open to any alternative ideas, though I have tried scanning ports and running TCP traceroutes to no avail. Note: this is a Windows environment so Linux is out of the question, sadly. Show me what you've got!

Python (but also C) Strange sorting done by gethostbyname

I have found this problem in Python but I was also able to reproduce it with a basic C program.
I am in CentOS 6 (tested also on 7), I have not tested on other Linux distributions.
I have an application on 2 VMs. One has IP address 10.0.13.30 and the other is 10.0.13.56. They have a shared FQDN to allow load-balancing (and high availability) DNS based using gethostbyname or getaddrinfo (it is what is suggested in Python doc).
If my client application is on a different sub-net (10.0.12.x for example), I have no problem: the socket.gethostbyname(FQDN) is returning randomly 10.0.13.30 and 10.0.13.56.
But if my client application is on the same sub-network, it returns always the same entry. And it seems to be always the "closest": I have depoyed it on 10.0.13.31 and it returns always 10.0.13.30 and on 10.0.13.59 it returns always 10.0.13.56.
On these servers CLI commands such as ping and dig are returning the result in different orders almost each time
I have searched many subjects and I concluded that it seems to be a kind of "prioritization to improve the success chances done by glibc" but I have not found any way to disable it.
Because clearly in my case the 2 clients and the 2 servers VMs are on the VMware connected to a single router, so I do not see how the fact that the last byte of the IP of the server is closest to the last byte of the IP of the client is taken into account.
This is a replication of a problem that I have at customer side so it is not an option for me to just move the VMs to a different sub-net :-( ....
Anybody has an idea to have correct load-balancing in the same sub-network? I can partially control the VM config so if a settings has to be changed I can do it.

Instead of hoping that the standard library will do load balancing for you, use socket.getaddrinfo() and choose one of the resulting hosts at random explicitly. This will also make it easy to fail over to a different host if the first one you try is not available.

Serial Communication one to one

If this is a stupid question, please don't mind me. But I spent some time trying to find the answer but I couldn't get anything solid. Maybe this is a hardware question, but I figured I'd try here first.
Does Serial Communication only work one to one? The reason this came up is because I had an arduino board listening for communication on its serial port. I had a python script feed bytes to the port as well. However, whenever I opened up the arduino's serial monitor, the connection with the python script failed. The serial monitor also connects to the serial port for communication for its little text input field.
So what's the deal? Does serial communication only work between a single client and a single server? Is there a way to get multiple clients writing to the server? I appreciate your suggestions.

Multiple clients (e.g. Arduinos) communicating with one server (e.g. a desktop computer) is commonly done with the serial variant:
RS-485
This is a simple method widely used in industrial settings where you want to have many devices connected to one computer via one serial port. This type of arrangement is also called multi-drop, because one cable strings around a building with Tees that tap in and drop lines to each device.
The hardware for this is widely available. You can buy USB serial adapters that provide the hardware interface for a computer. Programmatically the port looks just like an RS232 port. For the Arduino you would just add a transceiver chip. A sea of serial transceivers exists, e.g.
Example computer USB adapter with 485 interface
Sample RS485 transceiver chip from Element14
All the devices hang on the same bus listening at the same time. A simple communication protocol used is just add a device address before every command. For example:
001SETLIGHT1 <- tells Arduino "001" to turn on the light
013SETLIGHT0 <- tells "013" to turn off the light
Any device hanging on the cable ignores commands that do not start with their address. When a device responds, it prepends its address.
001SETLIGHT1DONE <- response from device "001" that the command has been received and executed
The address in the response lets the receiving party know which device was talking.

Well, your question can be quite wide, so I'm going to layer my answer:
On the hardware side, the same pair of wires can work be shared with many devices. It is mostly a question of electronics (maintaining the signal in the good voltage range), and not having all devices writing to the serial port at the same time (or you'll get wreckage).
On the software side, on the host, yes you can share the same serial connection to a device with multiple processes. But that's not straight forward. I'll assume you're using an unix (macos or linux):
in unix, everything is a file, your serial connection is one too: /dev/ttyACM0 on linux, for example.
When you have a process opening that file, it will block it (using ioctl, iirc) so no other process can mess with that file too.
Then, you can input and output to that file using the process that opened it, that's all.
But hopefully, it is still possible to share the connection between processes. One of them would simply be to use the tee command, that will be able to get input from one process, and give it back output, and copy the output to another process. You can also do it from within python, by duplicating the file descriptor.
To easily output stuff that can be redirected the unix way (using pipes), you can use socat: http://www.dest-unreach.org/socat/
here's an usage example:
socat -,raw,echo=0,escape=0x0f /dev/ttyACM0,raw,echo=0,crnl
you may want to tweak it for your needs.

Edit:
I forgot about RS-485, which 'jdr5ca' was smart enough to recommend. My explanation below is restricted to RS-232, the more "garden variety" serial port. As 'jdr5ca' points out, RS-485 is a much better alternative for the described problem.
Original:
To expand on zmo's answer a bit, it is possible to share serial at the hardware level, and it has been done before, but it is rarely done in practice.
Likewise, at the software driver level, it is again theoretically possible to share, but you run into similar problems as the hardware level, i.e. how to "share" the link to prevent collisions, etc.
A "typical" setup would be two serial (hardware) devices attached to each other 1:1. Each would run a single software process that would manage sending/receiving data on the link.
If it is desired to share the serial link amongst multiple processes (on either side), the software process that manages the link would also need to manage passing the received data to each reading process (keeping track of which data each process had read) and also arbitrate which sending process gets access to the link during "writes".
If there are multiple read/write processes on each end of the link, the handshaking/coordination of all this gets deep as some sort of meta-signaling arrangement may be needed to coordinate the comms between the process on each end.
Either a real mess or a fun challenge, depending on your needs and how you view such things.

MAC Address from IP across network

I hope you're all well.
I'm wondering if you could help me or point me in the right direction. I'm currently working on a project that centers around network management. Due to severe time constraints where possible I'm using opensource code. The issue I'm having is that part of the project requires me to be able to capture the MAC addresses of all of the devices that are connected to the network.
My knowledge of network orientated programming is limited as I have been working in other areas of software engineering for the past 4 years. The approach I have taken is to use nmap as a basis to get the ip address and other information I need. The MAC address is not included in the nmap out put and from what I have read it seems to be a bit flakey. (i could be wrong).
Therefore I have tried to do this in a two stage approach, firstly I get the data including ip address from nmap which works fine. My next step and the bit I'm having difficulty with is I ping the ip address (from within my python program) which works. But how do I get the MAC Address from the IP address? I initially thought ping the ip and grab the MAC from the ARP but I think that will only work if the IP address is on the same subnet. to compound the problem on deployment there could be up to 5000 computers on the network that needs to be logged. To show you my python ping approach this is the code:
import pdb, os
import subprocess
import re
from subprocess import Popen, PIPE
# This will only work within the netmask of the machine the program is running on cross router MACs will be lost
ip ="192.168.0.4"
#PING to place target into system's ARP cache
process = subprocess.Popen(["ping", "-c","4", ip], stdout=subprocess.PIPE)
process.wait()
result = process.stdout.read()
print(result)
#MAC address from IP
pid = Popen(["arp", "-n", ip], stdout=PIPE)
s = pid.communicate()[0]
# [a-fA-F0-9] = find any character A-F, upper and lower case, as well as any number
# [a-fA-F0-9]{2} = find that twice in a row
# [a-fA-F0-9]{2}[:|\-] = followed by either a ?:? or a ?-? character (the backslash escapes the hyphen, since the # hyphen itself is a valid metacharacter for that type of expression; this tells the regex to look for the hyphen character, and ignore its role as an operator in this piece of the expression)
# [a-fA-F0-9]{2}[:|\-]? = make that final ?:? or ?-? character optional; since the last pair of characters won't be followed by anything, and we want them to be included, too; that's a chunk of 2 or 3 characters, so far
# ([a-fA-F0-9]{2}[:|\-]?){6} = find this type of chunk 6 times in a row
mac = re.search(r"([a-fA-F0-9]{2}[:|\-]?){6}", s).groups()[0] #LINUX VERSION ARP
mac = re.search(r"(([a-f\d]{1,2}\:){5}[a-f\d]{1,2})", s).groups()[0] #MAC VERSION ARP
print(mac)
I have looked for some information but what I have found seems a bit vague. If you know of any ideas or avenues of research that may help me I would be greatful
Cheers
Chris

You can't directly get the MAC address of a machine outside your subnet.
A common strategy for network management applications is to query machines that do have this information, such as the routers and switches connecting the machines, using SNMP. Routers have arp tables for the subnets to which they are directly connected (as they need this to do their job), and this information can be acquired from the router.
The answers to this question might help with finding python library code to help in this endeavor.

You cannot get the original MAC address of the host if you are not connected to the same subnet - you would just get the MAC address of the last router.
The only way to get all MAC addresses would be to setup a server to catch them in each subnet, but this seems to me a bit crazy idea. Note also that nowadays it is very easy to spoof the MAC address, and it is not reliable at all.
Over all, I think you should use a different approach; for instance, there are plenty of network inventory systems, you could just use one of them, and interface with it.

You can use python scapy module to get mac address
from scapy.all import *
def get_mac(ip_address):
responses,unanswered = srp(Ether(dst="ff:ff:ff:ff:ff:ff")/ARP(pdst=ip_address),timeout=2,retry=10)
# return the MAC address from a response
for s,r in responses:
return r[Ether].src
return None
get_mac("192.168.31.14")