Develop Reference

Python - transfer files in the same network using shutil

I'm moving a .txt file between two servers (Ubuntu and Windows) in the same network.
The following code doesn't show any errors but it doesn't work:
def transfer_files_task():
source_path = r"/root/airflow/testdoc"
dest_path = f"192.168.xxx.xx\Doc-Share\Logger Output"
filename = r"/test.txt"
filenamew = f"\test.txt"
shutil.copyfile(source_path + filename, dest_path + filenamew)

Change your function to this:
import os, ntpath, posixpath
def transfer_files_task():
source_file = posixpath.join("/", "root", "airflow", "testdoc", "test.txt")
dest_file = ntpath.join("192.168.xxx.xx", "Doc-Share", "Logger Output", "test.txt")
assert os.path.exists(source_file), f"{source_file} does not exists"
shutil.copyfile(source_file, dest_file)
A small explanation: leave python format your paths, it will save you from many errors. If you do not do that, you have to know how strings work, what characters should be escaped, how to format a path on linux and windows, etc.
Also, a side note about the use of r and f prefix for strings:
r stands for raw and it roughly means that you don't have to escape special characters like the backspace. Consequently, r"\tab" == "\\tab" and print(r"\tab") gives \tab while print("\tab") gives ab
f stands for format and is the new way of formatting strings in py36+. It is used as follows:
name="john"
print(f"hello {name}")
# hello john
Finally, you might want to check this post: Cannot copy file from a remote machine using shutil

Merging multiple JSONs into one: TypeError, a bytes-like object is required, not 'str'

So, I am trying write a small program in python 3.6 to merge multiple JSON files (17k) and I am getting the above error.
I put together the script by reading other SO Q&As. I played around a little bit, getting various errors, nevertheless, I couldn't get it to work. Here is my code:
# -*- coding: utf-8 -*-
import glob
import json
import os
import sys
def merge_json(source, target):
os.chdir(source)
read_files = glob.glob("*.json")
result = []
i = 0
for f in glob.glob("*.json"):
print("Files merged so far:" + str(i))
with open(f, "rb") as infile:
print("Appending file:" + f)
result.append(json.load(infile))
i = i + 1
output_folder = os.path.join(target, "mergedJSON")
output_folder = os.path.join(output_folder)
if not os.path.exists(output_folder):
os.makedirs(output_folder)
os.chdir(output_folder)
with open("documents.json", "wb") as outfile:
json.dump(result, outfile)
try:
sys.argv[1], sys.argv[2]
except:
sys.exit("\n\n Error: Missing arguments!\n See usage example:\n\n python merge_json.py {JSON source directory} {output directory} \n\n")
merge_json(sys.argv[1], sys.argv[2])

In your case you are opening the file in 'wb' mode which means it works with bytes-like objects only. But json.dump is trying to write a string to it. You can simply change the open mode from 'wb' to 'w' (text mode) and it will work.

How to concatenate many binary file in one file in python?

I need to append many binary files in one binary file. All my binary files are saved i one folder:
file1.bin
file2.bin
...
For that I try by using this code:
import numpy as np
import glob
import os
Power_Result_File_Path ="/home/Deep_Learning_Based_Attack/Test.bin"
Folder_path =r'/home/Deep_Learning_Based_Attack/Test_Folder/'
os.chdir(Folder_path)
npfiles= glob.glob("*.bin")
loadedFiles = [np.load(bf) for bf in binfiles]
PowerArray=np.concatenate(loadedFiles, axis=0)
np.save(Power_Result_File_Path, PowerArray)
It gives me this error:
"Failed to interpret file %s as a pickle" % repr(file))
OSError: Failed to interpret file 'file.bin' as a pickle
My problem is how to concatenate binary file it is not about anaylysing every file indenpendently.

Taking your question literally: Brute raw data concatenation
files = ['my_file1', 'my_file2']
out_data = b''
for fn in files:
with open(fn, 'rb') as fp:
out_data += fp.read()
with open('the_concatenation_of_all', 'wb') as fp:
fp.write(out_data)
Comment about your example
You seem to be interpreting the files as saved numpy arrays (i.e. saved via np.save()). The error, however, tells me that you didn't save those files via numpy (because it fails decoding them). Numpy uses pickle to save and load, so if you try to open a random non-pickle file with np.load the call will throw an error.

for file in files:
async with aiofiles.open(file, mode='rb') as f:
contents = await f.read()
if file == files[0]:
write_mode = 'wb' # overwrite file
else:
write_mode = 'ab' # append to end of file
async with aiofiles.open(output_file), write_mode) as f:
await f.write(contents)

mmap write followed by flush does not seem to write to disk

I want the following to replace the binary string \x01\x02\x03 in the file test.bin with the binary string \x04\x05\x06. When I run the script it runs without error. However, when I run cmp -bl test.bin test.bin.old (where test.bin.old is an unmodified copy of the original file) then I get nothing (i.e. test.bin has not been modified). What's wrong please?
#!/usr/bin/python
import mmap
filename = "test.bin"
with open(filename, "r+b") as f:
mm = mmap.mmap(f.fileno(), 0, mmap.ACCESS_WRITE)
oldstr = '\x01\x02\x03'
foundpos = mm.find(oldstr)
if foundpos == -1:
print "Could not find string to replace"
exit(-1)
mm.seek(foundpos)
mm.write('\x04\x05\x06')
mm.flush()
mm.close()
f.close()
exit(0)

Error when trying to read and write multiple files

I modified the code based on the comments from experts in this thread. Now the script reads and writes all the individual files. The script reiterates, highlight and write the output. The current issue is, after highlighting the last instance of the search item, the script removes all the remaining contents after the last search instance in the output of each file.
Here is the modified code:
import os
import sys
import re
source = raw_input("Enter the source files path:")
listfiles = os.listdir(source)
for f in listfiles:
filepath = source+'\\'+f
infile = open(filepath, 'r+')
source_content = infile.read()
color = ('red')
regex = re.compile(r"(\b be \b)|(\b by \b)|(\b user \b)|(\bmay\b)|(\bmight\b)|(\bwill\b)|(\b's\b)|(\bdon't\b)|(\bdoesn't\b)|(\bwon't\b)|(\bsupport\b)|(\bcan't\b)|(\bkill\b)|(\betc\b)|(\b NA \b)|(\bfollow\b)|(\bhang\b)|(\bbelow\b)", re.I)
i = 0; output = ""
for m in regex.finditer(source_content):
output += "".join([source_content[i:m.start()],
"<strong><span style='color:%s'>" % color[0:],
source_content[m.start():m.end()],
"</span></strong>"])
i = m.end()
outfile = open(filepath, 'w+')
outfile.seek(0)
outfile.write(output)
print "\nProcess Completed!\n"
infile.close()
outfile.close()
raw_input()

The error message tells you what the error is:
No such file or directory: 'sample1.html'
Make sure the file exists. Or do a try statement to give it a default behavior.

The reason why you get that error is because the python script doesn't have any knowledge about where the files are located that you want to open.
You have to provide the file path to open it as I have done below. I have simply concatenated the source file path+'\\'+filename and saved the result in a variable named as filepath. Now simply use this variable to open a file in open().
import os
import sys
source = raw_input("Enter the source files path:")
listfiles = os.listdir(source)
for f in listfiles:
filepath = source+'\\'+f # This is the file path
infile = open(filepath, 'r')
Also there are couple of other problems with your code, if you want to open the file for both reading and writing then you have to use r+ mode. More over in case of Windows if you open a file using r+ mode then you may have to use file.seek() before file.write() to avoid an other issue. You can read the reason for using the file.seek() here.