I perform molecular dynamics wherein I process the output such that I get excel-like files as in:
-rw-r--r-- 1 staff 210030 Jan 20 16:25 summary.EKTOT
-rw-r--r-- 1 staff 210030 Jan 20 16:25 summary.EPTOT
-rw-r--r--# 1 staff 210030 Jan 20 16:25 summary.ETOT
-rw-r--r-- 1 staff 154022 Jan 20 16:25 summary.PRES
-rw-r--r--# 1 staff 105015 Jan 20 16:25 summary.DENSITY
-rw-r--r--# 1 staff 105015 Jan 20 16:25 summary.EKCMT
-rw-r--r-- 1 staff 105015 Jan 20 16:25 summary.TSOLUTE
-rw-r--r-- 1 staff 105015 Jan 20 16:25 summary.TSOLVENT
-rw-r--r-- 1 staff 105015 Jan 20 16:25 summary.VOLUME
I have downloaded these files to my computer with the goal of turning them all into csv/excel files such that the data can be entered and graphed into a jupyter notebook for record keeping. I have written the following code for this purpose (trying one file for the demo):
import pandas as pd
import os
os.chdir("Users/Downloads/analysis_BK_1")
with open(summary.DENSITY) as infile, open('summ_density.csv', 'w') as outfile:
for line in infile:
outfile.write(line.replace(' ', ','))
I keep getting the following error:
/Library/Frameworks/Python.framework/Versions/3.9/bin/python3 /Users/chrisgaughan/Desktop/Amber_Research_2020/excel_converter.py
(base) Chriss-MacBook-Pro:feature-engineering-for-machine-learning chris$ /Library/Frameworks/Python.framework/Versions/3.9/bin/python3 /Users/chrisgaughan/Desktop/Amber_Research_2020/excel_converter.py
Traceback (most recent call last):
File "/Users/Desktop/Amber_Research_2020/excel_converter.py", line 3, in <module>
os.chdir("Users/Downloads/analysis_BK_1")
FileNotFoundError: [Errno 2] No such file or directory: 'Users/chrisgaughan/Downloads/analysis_BK_1'
I have indeed checked that I am in the correct directory..if that really has anything to do with it.
Can someone please lend a hand?
Related
I am running a command line tool that returns a coloured output (similar to ls --color). I run this tool via subprocess:
process = subprocess.run(['ls --color'], shell=True, stdout=subprocess.PIPE)
process.stdout.decode()
But the result is, of course, with the color instructions like \x1b[m\x1b[m which makes further processing of the output impossible.
How can I remove the colouring and use pure text?
This works on my win10 and python 3.11 machine. Your command runs without any issue:
In my IDE I can't see color, but this command works also subprocess.run(["ls", "--color=none"], shell=True, stdout=subprocess.PIPE).
Valid arguments are on my machine:
'never', 'no', 'none'
'auto', 'tty', 'if-tty'
'always', 'yes', 'force'
Code:
import subprocess
process = subprocess.run(["ls", "-la"], shell=True, stdout=subprocess.PIPE)
with open("dirList.txt", 'w') as f:
f.write(process.stdout.decode())
Output:
total 14432
drwxr-xr-x 1 Hermann Hermann 0 Nov 19 08:00 .
drwxr-xr-x 1 Hermann Hermann 0 Aug 28 2021 ..
-rw-r--r-- 1 Hermann Hermann 0 Jan 28 09:25 .txt
-rw-r--r-- 1 Hermann Hermann 1225 Jan 6 00:51 00_Template_Program.py
-rw-r--r-- 1 Hermann Hermann 490 Jan 15 23:33 8859_1.py
-rw-r--r-- 1 Hermann Hermann 102 Jan 15 23:27 8859_1.xml
You example worked for me. You can also open the file with just 'w' and specify the encoding:
import subprocess
with open('output.txt', mode='w', encoding='utf-8') as file:
process = subprocess.run(['ls', '-l'], stdout=file)
My files in a directory are monthly data spanning several years, with characters like 0001-01-01, 0001-02-01, ..., 0005-01-01, ..., 0010-12-01 (yyyy-mm-dd) in the middle of each file name.
Now, I would like to exclude say the 0001* files.
If I wrote sorted(glob.glob(mydirectory/filename-000[!1]*)) only gives me 0002 ~~ 0009 files, while the 0010 files are not included.
What should I do to only exclude the 0001* files?
If I wrote sorted(glob.glob(mydirectory/filename-000[2-9]*)) also only gives me 0002-0009 files, what should I do to include the 0010* files?
I also tried filename-{000[2-9],00[10-12]}*, which does not work.
Thanks,
glob supports Unix shell pattern rules, but not more complex expressions. But, while you are in Python, you can use lots of filtering techniques, including regular expressions. If my directory looks like this:
$ find
.
./mydirectory
./mydirectory/filename-0001-01-01
./mydirectory/filename-0001-02-01
./mydirectory/filename-0010-12-01
./mydirectory/filename-0005-01-01
Then
[f for f in glob.glob(r"mydirectory/filename-*") if "0001" not in f]
will return:
['mydirectory/filename-0010-12-01', 'mydirectory/filename-0005-01-01']
This is further explained in this SO answer
Just add two globs together.
files = glob.glob(mydirectory/filename-000[!1]*) + glob.glob(mydirectory/filename-0010*)
Do it like this
import glob
from pprint import pprint
pprint(sorted(glob.glob("filename-???[!1]*")))
Where ? is "any character", like * is for "any characters string"
For me, this works pretty well.
(stackoverflow) ~/PycharmProjects/stackoverflow ls -la filename-00*
-rw-r--r-- 1 dude staff 0 Dec 11 23:31 filename-0001
-rw-r--r-- 1 dude staff 0 Dec 11 23:31 filename-0002
-rw-r--r-- 1 dude staff 0 Dec 11 23:31 filename-0003
-rw-r--r-- 1 dude staff 0 Dec 11 23:31 filename-0004
-rw-r--r-- 1 dude staff 0 Dec 11 23:31 filename-0005
-rw-r--r-- 1 dude staff 0 Dec 11 23:31 filename-0006
-rw-r--r-- 1 dude staff 0 Dec 11 23:31 filename-0007
-rw-r--r-- 1 dude staff 0 Dec 11 23:31 filename-0008
-rw-r--r-- 1 dude staff 0 Dec 11 23:31 filename-0009
-rw-r--r-- 1 dude staff 0 Dec 11 23:31 filename-0010
(stackoverflow) ~/PycharmProjects/stackoverflow # python test123.py
['filename-0002',
'filename-0003',
'filename-0004',
'filename-0005',
'filename-0006',
'filename-0007',
'filename-0008',
'filename-0009',
'filename-0010']
(stackoverflow) ~/PycharmProjects/stackoverflow #
I'm working on a project in which I've to do some serial communication with whichever device is connected (ttyS0, ttyS1 or ttyUSB0). Luckily I've come across a very useful stackoverflow link: "Simple way to query connected USB devices info in Python?". In this link there is a python code which works perfectly fine and it gives a proper device name and details.
here in the example code:
"/dev/bus/usb/005/002" is the device information of "FT232 Serial (UART)". SO, is there a way to find either mapping of /dev/bus/usb/005/002 with ttyS0/ ttyUSB0 or direct access of the UART with the device information and do the serial communication using "/dev/bus/usb/< bus >/< device >" instead of ttyS0 or ttyUSB0.
python code:
import re
import subprocess
device_re = re.compile("Bus\s+(?P<bus>\d+)\s+Device\s+(?P<device>\d+).+ID\s(?P<id>\w+:\w+)\s(?P<tag>.+)$", re.I)
df = subprocess.check_output("lsusb")
devices = []
for i in df.split('\n'):
if i:
info = device_re.match(i)
if info:
dinfo = info.groupdict()
dinfo['device'] = '/dev/bus/usb/%s/%s' % (dinfo.pop('bus'), dinfo.pop('device'))
devices.append(dinfo)
print devices
result:
{'device': '/dev/bus/usb/001/001', 'tag': 'Linux Foundation 2.0 root hub', 'id': '1d6b:0002'}
{'device': '/dev/bus/usb/005/002', 'tag': 'Future Technology Devices International, Ltd FT232 Serial (UART) IC', 'id': '0403:6001'}
{'device': '/dev/bus/usb/005/001', 'tag': 'Linux Foundation 1.1 root hub', 'id': '1d6b:0001'}
{'device': '/dev/bus/usb/004/003', 'tag': 'Lite-On Technology Corp. ', 'id': '04ca:0061'}
{'device': '/dev/bus/usb/004/002', 'tag': 'Dell Computer Corp. ', 'id': '413c:2107'}
{'device': '/dev/bus/usb/004/001', 'tag': 'Linux Foundation 1.1 root hub', 'id': '1d6b:0001'}
{'device': '/dev/bus/usb/003/001', 'tag': 'Linux Foundation 1.1 root hub', 'id': '1d6b:0001'}
{'device': '/dev/bus/usb/002/001', 'tag': 'Linux Foundation 1.1 root hub', 'id': '1d6b:0001'}
Thanking with regards
Aatif shaikh
A useful utilities is udevadm. For example, I have this USB serial adapter:
$ lsusb|grep -i prolific
Bus 001 Device 077: ID 067b:2303 Prolific Technology, Inc. PL2303 Serial Port
Running udevadm on it yields a bunch of information. Here's the start of it:
$ udevadm info -a /dev/bus/usb/001/077
Udevadm info starts with the device specified by the devpath and then
walks up the chain of parent devices. It prints for every device
found, all possible attributes in the udev rules key format.
A rule to match, can be composed by the attributes of the device
and the attributes from one single parent device.
looking at device '/devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3.4/1-3.4.4':
KERNEL=="1-3.4.4"
SUBSYSTEM=="usb"
DRIVER=="usb"
ATTR{authorized}=="1"
ATTR{avoid_reset_quirk}=="0"
ATTR{bConfigurationValue}=="1"
ATTR{bDeviceClass}=="00"
ATTR{bDeviceProtocol}=="00"
ATTR{bDeviceSubClass}=="00"
ATTR{bMaxPacketSize0}=="64"
ATTR{bMaxPower}=="100mA"
ATTR{bNumConfigurations}=="1"
ATTR{bNumInterfaces}==" 1"
ATTR{bcdDevice}=="0300"
ATTR{bmAttributes}=="a0"
ATTR{busnum}=="1"
ATTR{configuration}==""
ATTR{devnum}=="77"
ATTR{devpath}=="3.4.4"
ATTR{idProduct}=="2303"
ATTR{idVendor}=="067b"
ATTR{ltm_capable}=="no"
ATTR{manufacturer}=="Prolific Technology Inc."
ATTR{maxchild}=="0"
ATTR{product}=="USB-Serial Controller"
ATTR{quirks}=="0x0"
ATTR{removable}=="unknown"
ATTR{speed}=="12"
ATTR{urbnum}=="22"
ATTR{version}==" 2.00"
You can then look into sysfs for more details:
$ ls -l '/sys//devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3.4/1-3.4.4'
total 0
drwxr-xr-x 7 root root 0 Oct 11 11:03 1-3.4.4:1.0
-r--r--r-- 1 root root 4096 Oct 11 11:03 bcdDevice
-rw-r--r-- 1 root root 4096 Oct 11 11:03 bConfigurationValue
-r--r--r-- 1 root root 4096 Oct 11 11:03 bDeviceClass
-r--r--r-- 1 root root 4096 Oct 11 11:03 bDeviceProtocol
-r--r--r-- 1 root root 4096 Oct 11 11:03 bDeviceSubClass
-r--r--r-- 1 root root 4096 Oct 11 11:03 bmAttributes
-r--r--r-- 1 root root 4096 Oct 11 11:03 bMaxPacketSize0
-r--r--r-- 1 root root 4096 Oct 11 11:03 bMaxPower
-r--r--r-- 1 root root 4096 Oct 11 11:03 bNumConfigurations
-r--r--r-- 1 root root 4096 Oct 11 11:03 bNumInterfaces
-r--r--r-- 1 root root 4096 Oct 11 11:03 busnum
-r--r--r-- 1 root root 4096 Oct 11 11:03 configuration
-r--r--r-- 1 root root 65553 Oct 11 11:03 descriptors
-r--r--r-- 1 root root 4096 Oct 11 11:03 dev
-r--r--r-- 1 root root 4096 Oct 11 11:03 devnum
-r--r--r-- 1 root root 4096 Oct 11 11:03 devpath
lrwxrwxrwx 1 root root 0 Oct 11 11:03 driver -> ../../../../../../../bus/usb/drivers/usb
drwxr-xr-x 3 root root 0 Oct 11 11:03 ep_00
-r--r--r-- 1 root root 4096 Oct 11 11:03 idProduct
-r--r--r-- 1 root root 4096 Oct 11 11:03 idVendor
-r--r--r-- 1 root root 4096 Oct 11 11:03 ltm_capable
-r--r--r-- 1 root root 4096 Oct 11 11:03 manufacturer
-r--r--r-- 1 root root 4096 Oct 11 11:03 maxchild
lrwxrwxrwx 1 root root 0 Oct 11 11:03 port -> ../1-3.4:1.0/1-3.4-port4
drwxr-xr-x 2 root root 0 Oct 11 11:03 power
-r--r--r-- 1 root root 4096 Oct 11 11:03 product
-r--r--r-- 1 root root 4096 Oct 11 11:03 quirks
-r--r--r-- 1 root root 4096 Oct 11 11:03 removable
--w------- 1 root root 4096 Oct 11 11:03 remove
-r--r--r-- 1 root root 4096 Oct 11 11:03 speed
lrwxrwxrwx 1 root root 0 Oct 11 11:03 subsystem -> ../../../../../../../bus/usb
-rw-r--r-- 1 root root 4096 Oct 11 11:03 uevent
-r--r--r-- 1 root root 4096 Oct 11 11:03 urbnum
-r--r--r-- 1 root root 4096 Oct 11 11:03 version
You'll notice a sub-directory (1-3.4.4:1.0) for each implemented USB device(interface? function?) within the adapter (in my case just one; I have other 4-port USB serial adapter with 4 sub-directories too). If you look in there, you can eventually find the device node:
$ ls -l '/sys//devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3.4/1-3.4.4/1-3.4.4:1.0/'
total 0
-rw-r--r-- 1 root root 4096 Oct 11 11:03 authorized
-r--r--r-- 1 root root 4096 Oct 11 11:03 bAlternateSetting
-r--r--r-- 1 root root 4096 Oct 11 11:03 bInterfaceClass
-r--r--r-- 1 root root 4096 Oct 11 11:03 bInterfaceNumber
-r--r--r-- 1 root root 4096 Oct 11 11:03 bInterfaceProtocol
-r--r--r-- 1 root root 4096 Oct 11 11:03 bInterfaceSubClass
-r--r--r-- 1 root root 4096 Oct 11 11:03 bNumEndpoints
lrwxrwxrwx 1 root root 0 Oct 11 11:03 driver -> ../../../../../../../../bus/usb/drivers/pl2303
drwxr-xr-x 3 root root 0 Oct 11 11:03 ep_02
drwxr-xr-x 3 root root 0 Oct 11 11:03 ep_81
drwxr-xr-x 3 root root 0 Oct 11 11:03 ep_83
-r--r--r-- 1 root root 4096 Oct 11 11:03 modalias
drwxr-xr-x 2 root root 0 Oct 11 11:03 power
lrwxrwxrwx 1 root root 0 Oct 11 11:03 subsystem -> ../../../../../../../../bus/usb
-r--r--r-- 1 root root 4096 Oct 11 11:03 supports_autosuspend
drwxr-xr-x 4 root root 0 Oct 11 11:03 ttyUSB0
-rw-r--r-- 1 root root 4096 Oct 11 11:03 uevent
$ ls -l '/sys//devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3.4/1-3.4.4/1-3.4.4:1.0/ttyUSB0'
total 0
lrwxrwxrwx 1 root root 0 Oct 11 11:03 driver -> ../../../../../../../../../bus/usb-serial/drivers/pl2303
-r--r--r-- 1 root root 4096 Oct 11 11:03 port_number
drwxr-xr-x 2 root root 0 Oct 11 11:03 power
lrwxrwxrwx 1 root root 0 Oct 11 11:03 subsystem -> ../../../../../../../../../bus/usb-serial
drwxr-xr-x 3 root root 0 Oct 11 11:03 tty
-rw-r--r-- 1 root root 4096 Oct 11 11:03 uevent
$ ls -l '/sys//devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3.4/1-3.4.4/1-3.4.4:1.0/ttyUSB0/tty'
total 0
drwxr-xr-x 3 root root 0 Oct 11 11:11 ttyUSB0
$ ls -l '/sys//devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3.4/1-3.4.4/1-3.4.4:1.0/ttyUSB0/tty/ttyUSB0'
total 0
-r--r--r-- 1 root root 4096 Oct 11 11:11 dev
lrwxrwxrwx 1 root root 0 Oct 11 11:11 device -> ../../../ttyUSB0
drwxr-xr-x 2 root root 0 Oct 11 11:11 power
lrwxrwxrwx 1 root root 0 Oct 11 11:11 subsystem -> ../../../../../../../../../../../class/tty
-rw-r--r-- 1 root root 4096 Oct 11 11:11 uevent
$ cat '/sys//devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3.4/1-3.4.4/1-3.4.4:1.0/ttyUSB0/tty/ttyUSB0/dev'
188:0
That last line is the device node major/minor number, which you can search for in /dev, or use to create your own device node.
The directory /sys/class/tty/ lists all ttys, and /sys/bus/usb-serial/devices/ lists all USB serial adapters. It might be easier to start from those directories (rather than udevadm) to find all relevant devices, then determine which one of those matches the USB device you care about.
$ ls -l /sys/bus/usb-serial/devices/
total 0
lrwxrwxrwx 1 root root 0 Oct 11 11:16 ttyUSB0 -> ../../../devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3.4/1-3.4.4/1-3.4.4:1.0/ttyUSB0/
lrwxrwxrwx 1 root root 0 Sep 6 13:48 ttyUSB1 -> ../../../devices/pci0000:00/0000:00:14.0/usb1/1-4/1-4:1.1/ttyUSB1/
lrwxrwxrwx 1 root root 0 Oct 11 11:16 ttyUSB10 -> ../../../devices/pci0000:00/0000:00:14.0/usb1/1-8/1-8.4/1-8.4:1.3/ttyUSB10/
lrwxrwxrwx 1 root root 0 Sep 6 13:48 ttyUSB12 -> ../../../devices/pci0000:00/0000:00:14.0/usb1/1-8/1-8.6/1-8.6:1.0/ttyUSB12/
lrwxrwxrwx 1 root root 0 Sep 6 13:48 ttyUSB13 -> ../../../devices/pci0000:00/0000:00:14.0/usb1/1-8/1-8.6/1-8.6:1.1/ttyUSB13/
lrwxrwxrwx 1 root root 0 Sep 6 13:48 ttyUSB14 -> ../../../devices/pci0000:00/0000:00:14.0/usb1/1-8/1-8.6/1-8.6:1.2/ttyUSB14/
lrwxrwxrwx 1 root root 0 Sep 6 13:48 ttyUSB15 -> ../../../devices/pci0000:00/0000:00:14.0/usb1/1-8/1-8.6/1-8.6:1.3/ttyUSB15/
lrwxrwxrwx 1 root root 0 Sep 6 13:48 ttyUSB2 -> ../../../devices/pci0000:00/0000:00:14.0/usb1/1-4/1-4:1.2/ttyUSB2/
lrwxrwxrwx 1 root root 0 Sep 6 13:48 ttyUSB3 -> ../../../devices/pci0000:00/0000:00:14.0/usb1/1-4/1-4:1.3/ttyUSB3/
lrwxrwxrwx 1 root root 0 Oct 11 11:16 ttyUSB4 -> ../../../devices/pci0000:00/0000:00:14.0/usb1/1-8/1-8.4/1-8.4:1.0/ttyUSB4/
lrwxrwxrwx 1 root root 0 Sep 6 13:48 ttyUSB5 -> ../../../devices/pci0000:00/0000:00:14.0/usb1/1-7/1-7:1.1/ttyUSB5/
lrwxrwxrwx 1 root root 0 Sep 6 13:48 ttyUSB6 -> ../../../devices/pci0000:00/0000:00:14.0/usb1/1-7/1-7:1.2/ttyUSB6/
lrwxrwxrwx 1 root root 0 Sep 6 13:48 ttyUSB7 -> ../../../devices/pci0000:00/0000:00:14.0/usb1/1-7/1-7:1.3/ttyUSB7/
lrwxrwxrwx 1 root root 0 Oct 11 11:16 ttyUSB8 -> ../../../devices/pci0000:00/0000:00:14.0/usb1/1-8/1-8.4/1-8.4:1.1/ttyUSB8/
lrwxrwxrwx 1 root root 0 Oct 11 11:16 ttyUSB9 -> ../../../devices/pci0000:00/0000:00:14.0/usb1/1-8/1-8.4/1-8.4:1.2/ttyUSB9/
If you assume that you can take those path names (the link values) and go up to ../.. and find the USB device, and find its bus/dev number. I /think/ this is safe, but it depends on whether different USB devices/drivers always lay out their sysfs identically, which I believe they do for a given device type.
$ cat '/sys/devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3.4/1-3.4.4/1-3.4.4:1.0/ttyUSB0/../../busnum'
1
$ cat '/sys/devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3.4/1-3.4.4/1-3.4.4:1.0/ttyUSB0/../../devnum'
77
In general, sysfs contains all the information you want; you just have to work out how to traverse it.
Re: your comment on your question of "I'm just trying to find the link between 'device': '/dev/bus/usb/005/002', 'tag': 'Future Technology Devices International, Ltd FT232 Serial (UART) IC', 'id': '0403:6001'} and ttyUSB0. so that I can always check what type of device is connected to any serial port before I begin the serial communication.":
Be careful not to read too much into the device name or USB device/vendor ID or name. Here's why.
Device node name/path:
- There are different types (protocols) of USB serial adapters. For example, I have both /dev/ttyUSB0 and /dev/ttyACM0 on my system for different types of adapter.
- The user can use udev rules to rename or symlink to the device node. For example, on my system, I have a rule that created /dev/console-nano which is a symlink to some ttyUSB node such as /dev/ttyUSB0 (the exact value depends on the order the devices are enumerated).
Thus, you should only check (via sysfs) whether the device node is a tty and perhaps also whether it's backed by the USB subsystem, or similar subsystems such as ISA/PCI/PCIe serial ports.
USB vendor/device ID:
There are many many vendors of USB serial adapters, even just at the chip level. For example, FTDI (many devices e.g. FT232R, FT4232H, ...), CH430, PL2303, Linux USB device port (common on embedded hardware) acting as a CDC ACM serial device, etc.
Thus, you should detect devices based on their capability (being a tty or serial port), rather than based on their specific USB device/vendor ID.
the dir with the needed import is in PYTHONPATH
I've put a print in the top of init.py
import sys
print(sys.path)
and all is well - see 3rd line of code below
the dir with the needed import is in PYTHONPATH
I've put a print in the top of init.py
import sys
print(sys.path)
and all is well
Here's the folder structure:
dgl-home
--dglLib # the class lib that's not being found; has init.py
--dgl-env # pipenv venv
--flask-projects
----learn # the project in question
Here's console:
(dgl-env-nWAMFiVe) (xenial)les#localhost:~/dgl-home/dgl-env/flask-
projects/learn$ python run.py
sys.path ['/home/les/dgl-home/dgl-env/flask-projects/learn',
'/home/les/dgl-home/dgl-env/dglLib',
'/home/les/.local/share/virtualenvs/dgl-env-
nWAMFiVe/lib/python36.zip', '/home/les/.local/share/virtualenvs/dgl-
env-nWAMFiVe/lib/python3.6', '/home/les/.local/share/virtualenvs/dgl-
env-nWAMFiVe/lib/python3.6/lib-dynload', '/usr/lib/python3.6',
'/home/les/.local/share/virtualenvs/dgl-env-
nWAMFiVe/lib/python3.6/site-packages']
Traceback (most recent call last):
File "run.py", line 1, in
from learn import app
File "/home/les/dgl-home/dgl-env/flask-
projects/learn/learn/init.py", line 8, in
from dglLib import (
ModuleNotFoundError: No module named 'dglLib'
(dgl-env-nWAMFiVe) (xenial)les#localhost:~/dgl-home/dgl-env/flask-
projects/learn$ ls -alt /home/les/dgl-home/dgl-env/dglLib
total 72
drwxr-xr-x 7 les root 4096 Jun 20 18:07 ..
drwxr-xr-x 3 les les 4096 Jun 20 17:34 .
-rw-rw-r-- 1 les les 0 Jun 20 17:34 init.py
drwxrwxr-x 8 les les 4096 Jun 20 17:23 .git
-rw-rw-r-- 1 les les 4315 Jun 12 19:13 dglPickleToS3BucketClasses.py
-rw-rw-r-- 1 les les 10487 Jun 12 19:07 dglContactsClasses.py
-rw-rw-r-- 1 les les 3343 Jun 11 17:38 gaicClasses.html
-rw-rw-r-- 1 les les 5713 Jun 11 17:32
dglPickleToS3BucketClasses.html
-rw-rw-r-- 1 les les 22042 Jun 11 17:23 dglContactsClasses.html
-rw-rw-r-- 1 les les 784 Jun 10 00:24 gaicClasses.py
Found it -
https://python-notes.curiousefficiency.org/en/latest/python_concepts/import_traps.html#the-double-import-trap -
I put full path on PYTHONPATH - including the dir with the class defs - had to backup one level
I'm a beginner for deep-learning world. That's why I'm testing sample code for classifying image.
However, I got error message like as MemoryError soon after executing train.py.
This sample code is from below URL.
https://medium.com/#ageitgey/machine-learning-is-fun-part-3-deep-learning-and-convolutional-neural-networks-f40359318721
How can I solve this problem?
Thanks in advance.
steven#steven-VirtualBox:~/work/tflearn$ ll
total 5794836
drwxr-xr-x 3 steven steven 4096 3월 9 15:21 ./
drwxr-xr-x 6 steven steven 4096 2월 27 17:16 ../
-rw-r--r-- 1 steven steven 12288 2월 28 11:56 .full_dataset.pkl.swp
-rw-r--r-- 1 steven steven 147 2월 27 17:20 checkpoint
-rw-r--r-- 1 steven steven 61214 2월 27 17:52 convnet_cifar10.py
-rw-rw-r-- 1 steven steven 450189480 2월 28 10:32 data.zip
-rw-rw-r-- 1 steven steven 5483602303 6월 13 2016 full_dataset.pkl
drwxr-xr-x 2 steven steven 4096 2월 27 17:19 mnist/
-rw-r--r-- 1 steven steven 2884 2월 27 17:47 train.py
-rw-r--r-- 1 steven steven 2611 2월 28 11:50 train2.py
-rw-r--r-- 1 steven steven 2483 2월 27 17:19 weights_persistence.py
steven#steven-VirtualBox:~/work/tflearn$
steven#steven-VirtualBox:~/work/tflearn$ python train.py
Traceback (most recent call last):
File "train.py", line 20, in <module>
X, Y, X_test, Y_test = pickle.load(open("full_dataset.pkl", "rb"))
MemoryError
My pc's specification is as below.
OS : Ubuntu 64bit
CPU : Intel(R) Core(TM) i5-7200U CPU # 2.50GHz
Memory : 11350MB
You probably don't have enough RAM to open the file, pickle is known for requiring a 2-3 times more RAM than the size of the data.
I am not sure that it would help but try:
Instead of this:
X, Y, X_test, Y_test = pickle.load(open("full_dataset.pkl", "rb"))
Try:
with open("full_dataset.pkl", "rb") as file_object:
X, Y, X_test, Y_test = pickle.load(file_object)
#Do further Stuff