I have method in my tkinter application. This method is used to export data into CSV from another application. The loop for export the data is very heavy. It takes days of time to complete.
I just came across the multi threading concept. Its kind of difficult to understand, I spent an entire day on this but couldn't achieve anything. Below is the code I use in my loop. Can this be handled by multiple threads without freezing my tkinter UI?
I have a Label that shows the number of records(cells) exported, in the tkinter window.
def export_cubeData(self):
exportPath = self.entry_exportPath.get()
for b in itertools.product(*(k.values())):
self.update()
if (self.flag == 0):
list1 = list()
for pair in zip(dims, b):
list1.extend(pair)
list1.append(self.box_value.get())
mdx1 = mdx.format(*temp, *list1)
try:
data = tm1.cubes.cells.execute_mdx(mdx1)
data1 = Utils.build_pandas_dataframe_from_cellset(data)
final_df = final_df.append(data1)
cellCount = tm1.cubes.cells.execute_mdx_cellcount(mdx1)
finalcellCount = finalcellCount + cellCount
self.noOfRecordsProcessed['text'] = finalcellCount
except:
pass
else:
tm.showinfo("Export Interrupted", "Data export has been cancelled")
return
final_df.to_csv(exportPath)
print(time.time() - start)
tm.showinfo("info", "Data export has been completed")
self.noOfRecordsProcessed['text'] = '0'
Related
I have designed an marine engine simulator - it's a program that sends some serial DATA using a certain protocol called "BlueVision". The data is encoded based on a header, a block number, a block type, actual data (indexed), a checksum and a footer. Due to the fact that if you want to change a value in the actual data - then the checksum changes - I designed an recalculation of the block with the correct checksum when the data is changed. I made a GUI that allows me to change the value of 2 data points in order to be able to test live. The problem is that I don't know how to use threads or subprocesses correctly and the window keeps on freezing. The application works - but works badly and I kept on searching for a similar issue - and did find some suggestions but I haven't been able to implement it. Can someone help? PS: the code is awkward at best - please don't judge - I know I could of made it way better - but this is the best I could do with the time I had. The issue might not be clear to you if you don't have a COM port where the program writes to.
# Blue Vision is a MTU serial communication protocol - based on HEX values grouped in BLOCKS
import tkinter as tk
from tkinter import ttk
import time
import serial
def split_by_n(seq, n):
while seq:
yield seq[:n]
seq= seq[n:]
def after(self, ms, func=None, *args):
"""Call function once after given time.
MS specifies the time in milliseconds. FUNC gives the
function which shall be called. Additional parameters
are given as parameters to the function call. Return
identifier to cancel scheduling with after_cancel."""
win = tk.Tk()
win.title("GUI - TEST VERSION")
win.geometry("750x250")
label = tk.Label(win, text = "Main Engine Overspeed")
label.pack()
v = tk.StringVar()
v.set('00')
c = tk.StringVar()
c.set("00")
def setText(word):
v.set(word)
a = ttk.Button(win, text ="01", command =lambda:setText("01"))
a.pack()
b = ttk.Button(win, text="00", command = lambda:setText("00"))
b.pack()
label1 = tk.Label(win, text ="Main Engine Speed")
label1.pack()
name_entry = tk.Entry(win, textvariable=c)
name_entry.pack()
def task():
MyVar = v.get()
priEngSp =c.get()
if len(priEngSp) == 0:
priEngSp = '00'
block_3 = 'FAF500030002000000290100000001000100000001000001000000000000000000000000000222AF5F'
block_4 = 'FAF500040003000001A000004650000047E00000000000000000000000000007EF4000083D6000000000000000000000000000000000000000000000012C000006D600000000000000000000278D00000000000000007FFFFFFF000000000001991500000000000000000016E36000000000000923D8000971F8000001F40000059F000026AC00002774000005800000251C00000580000027740000283C0000056200001D4C00001F400000061800000000000060FB00004650000036B000007D0000008CA0000006180000251C0000000000000000000000000000284800192D500017A6B00000051B0000251CFFFFFFA8000002580000044C000000FA0000000000000000000006770000CB200000D6D8000006770000CB200000D6D80000060600005DC000000000000027100000000000000000000000000000000000000000000003C2000061A8000000000000000000000000000000000000000000000000000000000000000000000000000000000000363300000EA6000249F0FFFFFB1E000F42400000000000000000000000000000000000000000000032D9AF5F'
block_5 = 'FAF5000500020000005600000000000000000000000000000000000000000000007F7F0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000034AAF5F'
block_6 = 'FAF5000600020000003D00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000234AF5F'
block_7 = 'FAF5000700030000008C000006280000064A0000064A0000068D0000066B0000068D0000068D000006AE0000000000000000000006AE000006070000060700000607000005E5000005A2000006070000064A00000000000000000000062A000006AE000005A20000350EFFFFF07C00003CDEFFFFE8AC00000000000000000000000000000000000012DEAF5F'
block_8 = 'FAF50008000300000070000000000000112900000000000000000000059C000027740000283C000047E000000000000000000000000000000000000000000000000000000000000000007FFFFFFF7FFFFFFF0000055100002CEC0000000000000000000000000000000000000DD1AF5F'
block_9 = 'FAF50009000200000020000000000000000000000000000000000000021AAF5F'
block_10 = 'FAF5000A0002000000260000000000000000000000000000000000000000000000000221AF5F'
block_11 = 'FAF5000B0003000000EC00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002E9AF5F'
block_12 = 'FAF5000C000200000045000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000242AF5F'
block_2 = 'FAF50002000200000074010001000000000000000000000000000000000000000000007F0000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002E9AF5F'
checksum = hex(0x00)
block_2split = list(split_by_n(block_2,2))
block_2split[10] = MyVar.upper() #Overspeed Alarm Position
for i in range(len(block_2split)-6):
checksum = hex(int(checksum, 16) + int(block_2split[i], 16))
checksum_string = str(checksum)
checksum_actvalue = checksum_string[2:]
checksum_long = checksum_actvalue.rjust(8,'0').upper()
checksum_split = list(split_by_n(checksum_long,2))
block_2split[len(block_2split)-3] = checksum_split[3]
block_2split[len(block_2split)-4] = checksum_split[2]
block_2split[len(block_2split)-5] = checksum_split[1]
block_2split[len(block_2split)-6] = checksum_split[0]
Block_2Output = ''.join(str(item) for item in block_2split)
iEngineSpeed = int(priEngSp,10)
hEngineSpeed = hex(iEngineSpeed * 10)
sEngineSpeed = str(hEngineSpeed)[2:].upper()
while (len(sEngineSpeed)<8):
sEngineSpeed = '0' + sEngineSpeed
block_4split = list(split_by_n(block_4,4))
sEngineSpeed_split = list(split_by_n(sEngineSpeed,4))
block_4split[5] = sEngineSpeed_split[0]
block_4split[6] = sEngineSpeed_split[1]
Block_4joint = ''.join(str(item) for item in block_4split)
Block_4joint_sp = list(split_by_n(Block_4joint,2))
checksumb4 = hex(0x00)
for i in range(len(Block_4joint_sp)-6):
checksumb4 = hex(int(checksumb4, 16) + int(Block_4joint_sp[i], 16))
checksumb4_string = str(checksumb4)
checksumb4_actvalue = checksumb4_string[2:]
checksumb4_long = checksumb4_actvalue.rjust(8,'0').upper()
checksumb4_split = list(split_by_n(checksumb4_long,2))
Block_4joint_sp[len(Block_4joint_sp)-3] = checksumb4_split[3]
Block_4joint_sp[len(Block_4joint_sp)-4] = checksumb4_split[2]
Block_4joint_sp[len(Block_4joint_sp)-5] = checksumb4_split[1]
Block_4joint_sp[len(Block_4joint_sp)-6] = checksumb4_split[0]
Block_4Output = ''.join(str(item) for item in Block_4joint_sp)
blocks = [Block_2Output, block_3, Block_4Output, block_5, block_6, block_7, block_8, block_9, block_10, block_11, block_12]
with serial.Serial('COM5', '9600') as ser: #you might wanna comment this part out
for block in blocks:
print(block)
ser.write(bytes.fromhex(block.strip())) #you might wanna comment this part out
time.sleep(1)
win.after(200, task)
win.after(200, task)
win.mainloop()
To avoid freezing, one of the way is to use thread:
from threading import Thread
...
def task():
while True:
MyVar = v.get()
...
blocks = [Block_2Output, block_3, Block_4Output, block_5, block_6, block_7, block_8, block_9, block_10, block_11, block_12]
with serial.Serial('COM5', '9600') as ser: #you might wanna comment this part out
for block in blocks:
print(block)
ser.write(bytes.fromhex(block.strip())) #you might wanna comment this part out
time.sleep(1)
time.sleep(0.2)
# run task() in a thread
Thread(target=task, daemon=1).start()
win.mainloop()
i have less than 3 months of python programming under my belt but basically, i have a program that pulls values from the CoinGecko API indefinitely and creates processes so the functions that pull the data can run independently from one another, but id like for it to break its loop after i press the specified hotkey which is set to 'q'. whenever i press the hotkey, nothing happens and the loop just keeps running. i've tried using the keyboard.read_key() function, but that just stops my program from running until i press the q button, which causes the program to run the loop once and then close. i have no idea why the is_pressed() function refuses to work and id like some help from more advanced people
Piece of Code in question:
from multiprocessing.dummy import freeze_support
from pycoingecko import CoinGeckoAPI
import time
from multiprocessing import Process
from multiprocessing import Pool
import multiprocessing
import keyboard as kb
import psutil
cg = CoinGeckoAPI()
class CGCoin:
def __init__(self, coinname, coinid):
self.coinname = coinname
self.coinid = coinid
def pulldata(self):
while True:
wishtoquit = False
if kb.is_pressed('Q'):
print('ending after this loop')
wishtoquit = True
timestarted = time.asctime()
self.prices = []
self.daychanges = []
self.volumes = []
self.marketcaps = []
self.weekchanges = []
self.highs = []
self.lows = []
self.times = []
print(f'starting {self.coinname} reading at {timestarted}')
loops = 0
maxloops = 2
while loops < maxloops:
time.sleep(15)
coin = cg.get_coin_by_id(f'{self.coinid}')
time.sleep(5)
coinvalues = coin.get('market_data')
coinprices = coinvalues.get('current_price')
coinvolumes = coinvalues.get('total_volume')
mrktcaps = coinvalues.get('market_cap')
dayhigh = coinvalues.get('high_24h')
daylow = coinvalues.get('low_24h')
daychangepercentage = coinvalues.get('price_change_percentage_24h')
weekchangepercentage = coinvalues.get('price_change_percentage_7d')
coinprice = coinprices.get('usd')
coinvolume = coinvolumes.get('usd')
coincap = mrktcaps.get('usd')
coindayhigh = dayhigh.get('usd')
coindaylow = daylow.get('usd')
timepulled = time.asctime()
self.prices.append(coinprice)
self.daychanges.append(daychangepercentage)
self.volumes.append(coinvolume)
self.marketcaps.append(coincap)
self.weekchanges.append(weekchangepercentage)
self.highs.append(coindayhigh)
self.lows.append(coindaylow)
self.times.append(timepulled)
loops = loops + 1
print(loops)
timeended = time.asctime()
})
print(f'stopping {self.coinname} reading at {timeended}')
if wishtoquit:
print('ending loops')
break
time.sleep(5)
bitcoin = CGCoin('Bitcoin', 'bitcoin')
ethereum = CGCoin('Ethereum', 'ethereum')
if __name__ == '__main__':
freeze_support()
btcpul = Process(target=bitcoin.pulldata, name=bitcoin.coinname)
btcpul.start()
if anyone has any ideas or fully-functional workarounds id really like to hear them. id be extremely grateful for any help recieved
It looks like PyPi keyboard needs root permissions on linux.
You could just do kb.on_press_key("p", lambda _: sys.exit(0)) and just do a sys.exit(0) to end the script.
If you're running this in the terminal you should just be able to press ctrl+c to interrupt its execution.
ref: How to detect key presses?
so I've been thinking about this for a couple days now and I cant figure it out, I've searched around but couldn't find the answer I was looking for, so any help would be greatly appreciated.
Essentially what I am trying to do is call a method on a group of objects in my main thread from a separate thread, just once after 2 seconds and then the thread can exit, I'm just using threading as a way of creating a non-blocking 2 second pause (if there are other ways of accomplishing this please let me know.
So I have a pyqtplot graph/plot that updates from a websocket stream and the gui can only be updated from the thread that starts it (the main one).
What happens is I open a websocket stream fill up a buffer for about 2 seconds, make an REST request, apply the updates from the buffer to the data from the REST request and then update the data/plot as new messages come in. Now the issue is I can't figure out how to create a non blocking 2 second pause in the main thread without creating a child thread. If I create a child thread and pass the object that contains the dictionary I want to update after 2 seconds, I get issues regarding updating the plot from a different thread. So what I THINK is happening is when that new spawned thread is spawned the reference to the object I want to update is actually the object itself, or the data (dictionary) containing the update data is now in a different thread as the gui and that causes issues.
open websocket --> start filling buffer --> wait 2 seconds --> REST request --> apply updates from buffer to REST data --> update data as new websocket updates/messages come in.
Unfortunately the websocket and gui only start when you run pg.exec() and you can't break them up to start individually, you create them and then start them together (or at least I have failed to find a way to start them individually, alternatively I also tried using a separate library to handle websockets however this requires starting a thread for incoming messages as well)
This is the minimum reproducible example, sorry it's pretty long but I couldn't really break it down anymore without removing required functionality as well as preserving context:
import json
import importlib
from requests.api import get
import functools
import time
import threading
import numpy as np
import pyqtgraph as pg
from pyqtgraph.Qt import QtCore
QtWebSockets = importlib.import_module(pg.Qt.QT_LIB + '.QtWebSockets')
class coin():
def __init__(self):
self.orderBook = {'bids':{}, 'asks':{}}
self.SnapShotRecieved = False
self.last_uID = 0
self.ordBookBuff = []
self.pltwgt = pg.PlotWidget()
self.pltwgt.show()
self.bidBar = pg.BarGraphItem(x=[0], height=[1], width= 1, brush=(25,25,255,125), pen=(0,0,0,0))
self.askBar = pg.BarGraphItem(x=[1], height=[1], width= 1, brush=(255,25,25,125), pen=(0,0,0,0))
self.pltwgt.addItem(self.bidBar)
self.pltwgt.addItem(self.askBar)
def updateOrderBook(self, message):
for side in ['a','b']:
bookSide = 'bids' if side == 'b' else 'asks'
for update in message[side]:
if float(update[1]) == 0:
try:
del self.orderBook[bookSide][float(update[0])]
except:
pass
else:
self.orderBook[bookSide].update({float(update[0]): float(update[1])})
while len(self.orderBook[bookSide]) > 1000:
del self.orderBook[bookSide][(min(self.orderBook['bids'], key=self.orderBook['bids'].get)) if side == 'b' else (max(self.orderBook['asks'], key=self.orderBook['asks'].get))]
if self.SnapShotRecieved == True:
self.bidBar.setOpts(x0=self.orderBook['bids'].keys(), height=self.orderBook['bids'].values(), width=1 )
self.askBar.setOpts(x0=self.orderBook['asks'].keys(), height=self.orderBook['asks'].values(), width=1 )
def getOrderBookSnapshot(self):
orderBookEncoded = get('https://api.binance.com/api/v3/depth?symbol=BTCUSDT&limit=1000')
if orderBookEncoded.ok:
rawOrderBook = orderBookEncoded.json()
orderBook = {'bids':{}, 'asks':{}}
for orders in rawOrderBook['bids']:
orderBook['bids'].update({float(orders[0]): float(orders[1])})
for orders in rawOrderBook['asks']:
orderBook['asks'].update({float(orders[0]): float(orders[1])})
last_uID = rawOrderBook['lastUpdateId']
while self.ordBookBuff[0]['u'] <= last_uID:
del self.ordBookBuff[0]
if len(self.ordBookBuff) == 0:
break
if len(self.ordBookBuff) >= 1 :
for eachUpdate in self.ordBookBuff:
self.last_uID = eachUpdate['u']
self.updateOrderBook(eachUpdate)
self.ordBookBuff = []
self.SnapShotRecieved = True
else:
print('Error retieving order book.') #RESTfull request failed
def on_text_message(message, refObj):
messaged = json.loads(message)
if refObj.SnapShotRecieved == False:
refObj.ordBookBuff.append(messaged)
else:
refObj.updateOrderBook(messaged)
def delay(myObj):
time.sleep(2)
myObj.getOrderBookSnapshot()
def main():
pg.mkQApp()
refObj = coin()
websock = QtWebSockets.QWebSocket()
websock.connected.connect(lambda : print('connected'))
websock.disconnected.connect(lambda : print('disconnected'))
websock.error.connect(lambda e : print('error', e))
websock.textMessageReceived.connect(functools.partial(on_text_message, refObj=refObj))
url = QtCore.QUrl("wss://stream.binance.com:9443/ws/btcusdt#depth#1000ms")
websock.open(url)
getorderbook = threading.Thread(target = delay, args=(refObj,), daemon=True) #, args = (lambda : websocketThreadExitFlag,)
getorderbook.start()
pg.exec()
if __name__ == "__main__":
main()
I'm very new to Python (with most of my previous programming experience being in intermediate C++ and Java) and am trying to develop a script which will read sensor data and log it to a .csv file. To do this I created separate classes for the code-- one will read the sensor data and output it to the console, while the other is supposed to take that data and log it-- and combined them together into a master script containing each class. Separately, they work perfectly, but together only the sensorReader class functions. I am trying to get each class to run in its own thread, while passing the sensor data from the first class (sensorReader) to the second class (csvWriter) as well. I've posted some of my pseudocode below, but I'd be happy to clarify any questions with the actual source code if needed.
import time
import sensorStuff
import csv
import threading
import datetime
class sensorReader:
# Initializers for the sensors.
this.code(initializes the sensors)
while True:
try:
this.code(prints the sensor data to the console)
this.code(throws exceptions)
this.code(waits 60 seconds)
class csvWriter:
this.code(fetches the date and time)
this.code(writes the headers for the excel sheet once)
while True:
this.code(gets date and time)
this.code(writes the time and one row of data to excel)
this.code(writes a message to console then repeats every minute)
r = sensorReader()
t = threading.Thread(target = r, name = "Thread #1")
t.start()
t.join
w = csvWriter()
t = threading.Thread(target = w, name = "Thread #2")
t.start()
I realize the last part doesn't really make sense, but I'm really punching above my weight here, so I'm not even sure why only the first class works and not the second, let alone how to implement threading for multiple classes. I would really appreciate it if anyone could point me in the right direction.
Thank you!
EDIT
I've decided to put up the full source code:
import time
import board
import busio
import adafruit_dps310
import adafruit_dht
import csv
import threading
import datetime
# import random
class sensorReader:
# Initializers for the sensors.
i2c = busio.I2C(board.SCL, board.SDA)
dps310 = adafruit_dps310.DPS310(i2c)
dhtDevice = adafruit_dht.DHT22(board.D4)
while True:
# Print the values to the console.
try:
global pres
pres = dps310.pressure
print("Pressure = %.2f hPa"%pres)
global temperature_c
temperature_c = dhtDevice.temperature
global temperature_f
temperature_f = temperature_c * (9 / 5) + 32
global humidity
humidity = dhtDevice.humidity
print("Temp: {:.1f} F / {:.1f} C \nHumidity: {}% "
.format(temperature_f, temperature_c, humidity))
print("")
# Errors happen fairly often with DHT sensors, and will occasionally throw exceptions.
except RuntimeError as error:
print("n/a")
print("")
# Waits 60 seconds before repeating.
time.sleep(10)
class csvWriter:
# Fetches the date and time for future file naming and data logging operations.
starttime=time.time()
x = datetime.datetime.now()
# Writes the header for the .csv file once.
with open('Weather Log %s.csv' % x, 'w', newline='') as f:
fieldnames = ['Time', 'Temperature (F)', 'Humidity (%)', 'Pressure (hPa)']
thewriter = csv.DictWriter(f, fieldnames=fieldnames)
thewriter.writeheader()
# Fetches the date and time.
while True:
from datetime import datetime
now = datetime.now()
current_time = now.strftime("%H:%M:%S")
# Writes incoming data to the .csv file.
with open('Weather Log %s.csv', 'a', newline='') as f:
fieldnames = ['TIME', 'TEMP', 'HUMI', 'PRES']
thewriter = csv.DictWriter(f, fieldnames=fieldnames)
thewriter.writerow({'TIME' : current_time, 'TEMP' : temperature_f, 'HUMI' : humidity, 'PRES' : pres})
# Writes a message confirming the data's entry into the log, then sets a 60 second repeat cycle.
print("New entry added.")
time.sleep(10.0 - ((time.time() - starttime) % 10.0)) # Repeat every ten seconds.
r = sensorReader()
t = threading.Thread(target = r, name = "Thread #1")
t.start()
t.join
w = csvWriter()
t = threading.Thread(target = w, name = "Thread #2")
t.start()
It would work better structured like this. If you put the first loop in a function, you can delay its evaluation until you're ready to start the thread. But in a class body it would run immediately and you never get to the second definition.
def sensor_reader():
# Initializers for the sensors.
this.code(initializes the sensors)
while True:
try:
this.code(prints the sensor data to the console)
except:
print()
this.code(waits 60 seconds)
threading.Thread(target=sensor_reader, name="Thread #1", daemon=True).start()
this.code(fetches the date and time)
this.code(writes the headers for the excel sheet once)
while True:
this.code(gets date and time)
this.code(writes the time and one row of data to excel)
this.code(writes a message to console then repeats every minute)
I made it a daemon so it will stop when you terminate the program. Note also that we only needed to create one thread, since we already have the main thread.
I'm really looking for a good solution here, maybe the complete concept how I did it or at elast tried to do it is wrong!?
I want to make my code capable of using all my cores. In the code I'm modifying Excel Cells using Win32 API. I wrote a small xls-Class which can check whether the desired file is already open (or open it if not so) and set Values to Cells. My stripped down code looks like this:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import os
import win32com.client as win32
from multiprocessing import Pool
from time import sleep
class xls:
excel = None
filename = None
wb = None
ws = None
def __init__(self, file):
self.filename = file
def getNumOpenWorkbooks(self):
return self.excel.Workbooks.Count
def openExcelOrActivateWb(self):
self.excel = win32.gencache.EnsureDispatch('Excel.Application')
# Check whether one of the open files is the desired file (self.filename)
if self.getNumOpenWorkbooks() > 0:
for i in range(self.getNumOpenWorkbooks()):
if self.excel.Workbooks.Item(i+1).Name == os.path.basename(self.filename):
self.wb = self.excel.Workbooks.Item(i+1)
break
else:
self.wb = self.excel.Workbooks.Open(self.filename)
def setCell(self, row, col, val):
self.ws.Cells(row, col).Value = val
def setLastWorksheet(self):
self.ws = self.wb.Worksheets(self.wb.Worksheets.Count)
if __name__ == '__main__':
dat = zip(range(1, 11), [1]*10)
# Create Object
xls = xls('blaa.xls')
xls.openExcelOrActivateWb()
xls.setLastWorksheet()
for (row, col) in dat:
# Calculate some value here (only depending on row,col):
# val = some_func(row, col)
val = 'test'
xls.setCell(row, col, val)
Now as the loop does ONLY depend on the both iterated vars, I wanted to make it run in parallel on many cores. So I've heard of Threading and Multiprocessing, but the latter seemed easier to me so I gave it a go.
So I changed the code like this:
import os
import win32com.client as win32
from multiprocessing import Pool
from time import sleep
class xls:
### CLASS_DEFINITION LIKE BEFORE ###
''' Define Multiprocessing Worker '''
def multiWorker((row, col)):
xls.setCell(row, col, 'test')
if __name__ == '__main__':
# Create Object
xls = xls('StockDatabase.xlsm')
xls.openExcelOrActivateWb()
xls.setLastWorksheet()
dat = zip(range(1, 11), [1]*10)
p = Pool()
p.map(multiWorker, dat)
Didn't seem to work because after some reading, Multiprocessing starts new Processes hence xls is not known to the workers.
Unfortunately I can neither pass xls to them as a third parameter as the Win32 can't be pickled :( Like this:
def multiWorker((row, col, xls)):
xls.setCell(row, col, 'test')
if __name__ == '__main__':
# Create Object
xls = xls('StockDatabase.xlsm')
xls.openExcelOrActivateWb()
xls.setLastWorksheet()
dat = zip(range(1, 11), [1]*10, [xls]*10)
p = Pool()
p.map(multiWorker, dat)
The only way would be to initialize the Win32 for each process right before the definition of the multiWorker:
# Create Object
xls = xls('StockDatabase.xlsm')
xls.openExcelOrActivateWb()
xls.setLastWorksheet()
def multiWorker((row, col, xls)):
xls.setCell(row, col, 'test')
if __name__ == '__main__':
dat = zip(range(1, 11), [1]*10, [xls]*10)
p = Pool()
p.map(multiWorker, dat)
But I don't like it because my constructor of xls has some more logic, which automatically tries to find column ids for known header substrings... So that is a little bit more effort then wanted (and I don't think each process should really open it's own Win32 COM Interface), and this also gives me an error because gencache.EnsureDispatch might not be possible to call so often....
What to do? How is the solution?
Thanks!!
While Excel can use multiple cores when recalculating spreadsheets, its programmatic interface is strongly tied to the UI model, which is single threaded. The active workbook, worksheet, and selection are all singleton objects; this is why you cannot interact with the Excel UI at the same time you're driving it using COM (or VBA, for that matter).
tl;dr
Excel doesn't work that way.