I have some VLP16 LiDar data in .csv file format, have to load the data in Ros Rviz for which I need the Rosbag file(.bag). I have tried finding it in the Ros tutorial, what I got was to convert .bag to .csv
I'm not actually expert in processing .bag files but I think you need to go through your CSV file and manually add the values using rosbag Python API
Not direct answer but check this script in python, which might help you.
Regarding C++ I propose this repository: convert_csv_to_rosbag which is even closer to what you asked.
However, it seems that you need to do it by yourself based on these examples.
Scenario: I just got my hands on a huge ntriples file (6.5gb uncompressed). I am trying to open it and perform some operations (such as cleaning some of the data that it contains).
Issue: I haven't been able to check the contents of this file. Notepad++ cannot handle it, and in RDFlib, the far as I got was to load the file, but I cannot seem to find a way to edit without parsing the entire thing. I also tried using RDF package (from how to parse big datasets using RDFLib?), but I cannot find a way to install it in Python 3.
Question: What is the best option to perform this kind of operation? Is there any command in rdflib that allows for this kind of editing?
if it's ntriples then basically it's a line-by-line triples. Therefore, you can read the file by small chunks (some N lines from the file) and parse the chunk via rdflib followed by any cleaning operation you need on the graph.
My friend has sent me the .bytes file from unity for machine learning model.
But I don't know how to open it in python for ML. Could any one able to give me the answer about the .bytes file in unity.
What is the purpose of it?
How to use it for python?
What is the purpose of it?
It's a raw bytes of a file. Let's say that you want to import a file into Unity Resources folder and the file is not one of the supported resources files in Unity such as .png, .mp3 and .mp4, but you want Unity to include this special file extension in the final build, you change the file extension to .bytes. It will be included in the final build and you can load and retrieve it during run-time with the Resources API and TextAsset.bytes. It's simply used to hold binary data.
How to use it for python?
You can like you would with any binary file in python.
byte = f.read(1)
binary_string = bin(int(binascii.hexlify(byte), 16))[2:].zfill(8)
See this for more information.
.bytes file has many usages like in unity
1: Used as the text asset
2: Used as graph model for the brain in ML
Now point is how to use it in python
1: bytes file is directly convert to the .pb file in python by just renaming.
for more details refer:
https://github.com/Unity-Technologies/ml-agents/issues/735
Now how to use it in python code:
https://gist.github.com/jubjamie/2eec49ca1e4f58c5310d72918d991ef6
I'd like to be able to read in the first couple kilobytes of unknown file types and see if it matches any known file types (i.e. mp3 file, jpeg, etc...). I was thinking of trying to load meta data from files from libraries like PIL, sndhdr, py264, etc... and see if they picked up any valid formats but I thought this must have been a problem someone has solved before.
Is there one library or a gist showing the usage of multiple libraries which would do this?
Use python-magic to do the fingerprinting.
The library can determine file type from bytes data only:
import magic
magic.from_buffer(start_data_from_something)
The library provides access to the libmagic file type identification library, which also drives the UNIX file command.
I am writing a Python script to index a large set of Windows installers into a DB.
I would like top know how to read the metadata information (Company, Product Name, Version, etc) from EXE, MSI and ZIP files using Python running on Linux.
Software
I am using Python 2.6.5 on Ubuntu 10.04 64-bit with Django 1.2.1.
Found so far:
Windows command line utilities that can extract EXE metadata (like filever from SysUtils), or other individual CL utils that only work in Windows. I've tried running these through Wine but they have problems and it hasn't been worth the work to go and find the libs and frameworks that those CL utils depend on and try installing them in Wine/Crossover.
Win32 modules for Python that can do some things but won't run in Linux (right?)
Secondary question:
Obviously changing the file's metadata would change the MD5 hashsum of the file. Is there a general method of hashing a file independent of the metadata besides locating it and reading it in (ex: like skipping the first 1024 byes?)
Take a look at this library: http://bitbucket.org/haypo/hachoir/wiki/Home and this example program that uses the library: http://pypi.python.org/pypi/hachoir-metadata/1.3.3. The second link is an example program which uses the Hachoir binary file manipulation library (first link) to parse the metadata.
The library can handle these formats:
Archives: bzip2, gzip, zip, tar
Audio: MPEG audio ("MP3"), WAV, Sun/NeXT audio, Ogg/Vorbis (OGG), MIDI, AIFF, AIFC, Real audio (RA)
Image: BMP, CUR, EMF, ICO, GIF, JPEG, PCX, PNG, TGA, TIFF, WMF, XCF
Misc: Torrent
Program: EXE
Video: ASF format (WMV video), AVI, Matroska (MKV), Quicktime (MOV), Ogg/Theora, Real media (RM)
Additionally, Hachoir can do some file manipulation operations which I would assume includes some primitive metadata manipulation.
The hachoir-metadata get the "Product Version" but the compilers changes the "File Version".
Then the version returned is not the we need.
I found a small a well working soluction:
http://pev.sourceforge.net/
I've tested with success. It's simple, fast and stable.
To answer one of your questions, you can use the zipfile module, specifically the ZipInfo object to get the metadata for zip files.
As for hashing only the data of the file, you can only to that if you know which parts are data and which are metadata. There can be no general method as many file formats store their metadata differently.
To answer your second question: no, there is no way to hash a PE file or ZIP file, ignoring the metadata, without locating and reading the metadata. This is because the metadata you're interested in is stored at variable locations in the file.
In the case of PE files (EXE, DLL, etc), it's stored in a resource block, typically towards the end of the file, and a series of pointers and tables at the start of the file gives the location.
In the case of ZIP files, it's scattered throughout the archive -- each included file is preceded by its own metadata, and then there's a table at the end giving the locations of each metadata block. But it sounds like you might actually be wanting to read the files within the ZIP archive and look for EXEs in there if you're after program metadata; the ZIP archive itself does not store company names or version numbers.