Develop Reference

Python: Search in an online CSV file

I am currently stuck with a Python project where I want to get the information out of an online CSV file and want to let the user search via an input function. At the moment, I am able to get the information from the online CSV file via a link but I cannot make the connection so that it searches the exact word in that CSV file.
I currently have tried multiple tutorials but most of them aren't solving my issue. So with a lot of pain, I am writing this message here, hoping someone can help me out.
The code I have so far is:
import csv
import urllib.request
metar_search = input('Enter the ICAO station\n')
url = 'https://www.aviationweather.gov/adds/dataserver_current/current/metars.cache.csv'
response = urllib.request.urlopen(url)
lines = [l.decode('utf-8') for l in response.readlines()]
cr = csv.reader(lines)
for row in cr:
if metar_search == row[0]:
print(row)
In the CSV file, the first row is what I am looking for. It is the METAR information of an airport. So, I want the user to type the ICAO code (for example KJFK), then I want the line of text the weather information of that station (example: KJFK 051851Z 15010KT 10SM FEW017 FEW035 FEW250 27/19 A3006 RMK AO2 SLP177 T02670194).
When I currently type KJFk, it is not returning any information back.
The current code is probably a bit messy because I have tried several things, I also tried to make a function of it but without luck. What am I doing wrong?
I hope someone is able to help me out with this question.
Thank you so much in advance.

Try
...
for row in cr:
if row[0].startswith(metar_search):
print(row)
or
...
lines = [l.decode('utf-8') for l in response.readlines()[5:]]
cr = csv.reader(lines)
for row in cr:
if metar_search == row[1]:
print(row)
Hint: Take a closer look at the data.
If you know that there's only one result then you could stop searching after you found the row:
...
print(row)
break

How to make an input function to find something in a certain row in a CSV file using Tkinter

Okay so I'm making a GUI to look for a specific number in a CSV file basically
ID: Name: Address: Email:
1023 John 123 Normal St John123#hotmail.com
So basically I want the person using the GUI to type in the ID and the GUI just goes through the CSV file and prints the whole row
Also I'm a novice coder so please don't judge me if I keep asking what a certain element is or what does this thing do
Thank you

First Off CSV stands for Comma Separated Values so naturally,we would expect your columns to be comma-separated not space -separated,I tried to adapt the code with your data but in the future,do separate your .csv data with commas
You could simply use readlines to read the contents of a .csv file.The logic you are looking for will be something like:
idx = 2710
with open('my.csv','r') as f:
data = f.readlines()
for d in data:
if d.find(str(idx)) != -1:
print(d)
break
i have already answered a similar question on the tkinter gui here,which applies to the same case as yours,all you have to do is paste the above logic into the on_click function and replace print(d) with text.insert(INSERT, d)
I would recommend reading about tkinter,also just a side dish pandas has some nice functions for working with .csv files,worth reading about

csv.writer returning long strings in each row

I've been having some more problems. After you've modified my code well into this.
import csv
mesta=["Ljubljana","Kranj","Skofja Loka","Trzin"]
opis=["ti","mene","ti mene","ne ti mene"]
delodajalci=["GENI","MOJEDELO","MOJADELNICA","HSE"]
ime=["domen","maja","andraz","sanja"]
datum=["2.1.2014","5.10.2014","11.12.2014","5.5.2014"]
with open('sth.csv','w') as csvfile:
zapis = csv.writer(csvfile)
zapis.writerows(zip(ime,delodajalci,opis,datum,mesta))
I have one aditional question. How do I get each piece of my output to have it's own cell and not have 5 really long rows divided by , signs. Since now my output looks like:
domen,GENI,ti,2.1.2014,Ljubljana
maja,MOJEDELO,mene,5.10.2014,Kranj
andraz,MOJADELNICA,ti mene,11.12.2014,Skofja Loka
sanja,HSE,ne ti mene,5.5.2014,Trzin
I hope you will be able to help me. Thank you in advance. Cheers.

So a csv file (Comma-separated values file) is meant to have commas on really long rows as you indicated. To open the file with each value in a cell, say for excel, if you change the extension of the file to .csv it will likely be taken care of. Otherwise, you may need to import the file and indicate that the separators are commas. If you don't have excel, you can try googling for csv viewer (there are many free versions available). In either case, your output looks correct, I think you just need a bit of help opening the file in your program of choice.

Building on "How to read and write a table / matrix to file with python?"

Back in Feb 8 '13 at 20:20, YamSMit asked a question (see: How to read and write a table / matrix to file with python?) similar to what I am struggling with: starting out with an Excel table (CSV) that has 3 columns and a varying number of rows. The contents of the columns are string, floating point, and string. The first string will vary in length, while the other string can be fixed (eg, 2 characters). The table needs to go into a 2 dimensional array, so that I can do manipulations on the data to produce a final file (which will be a text file). I have experimented with a variety of strategies presented in stackoverflow, but I am always missing something, and I haven't seen an example with all the parts, which is the reason for the struggle to figure this out.
Sample data will be similar to:
Ray Smith, 41645.87778, V1
I have read and explored numpy and astropy since the available documentation says they make this type of code easy. I have tried import csv. Somehow, the code doesn't come together. I should add that I am writing in Python 3.2.3 (which seems to be a mistake since a lot of documentation is for Python 2.x).
I realize the basic nature of this question directs me to read more tutorials. I have been reading many, yet the tutorials always refer to enough that is different, that I fail to assemble the right pieces: read the table file, write into a 2D array, then... do more stuff.
I am grateful to anyone who might provide me with a workable outline of the code, or pointing me to specific documentation I should read to handle the specific nature of the code I am trying to write.
Many thanks in advance. (Sorry for the wordiness - just trying to be complete.)

I am more familiar with 2.x, but from the 3.3 csv documentation found here, it seems to be mostly the same as 2.x. The following function will read a csv file, and return a 2D array of the rows found in the file.
import csv
def read_csv(file_name):
array_2D = []
with open(file_name, 'rb') as csvfile:
read = csv.reader(csvfile, delimiter=';') #Assuming your csv file has been set up with the ';' delimiter - there are other options, for which you should see the first link.
for row in read:
array_2D.append(row)
return array_2D
You would then be able to manipulate the data as follows (assuming your csv file is called 'foo.csv' and the desired text file is 'foo.txt'):
data = read_csv('foo.csv')
with open('foo.txt') as textwrite:
for row in data:
string = '{0} has {1} apples in his Ford {2}.\n'.format(row[0], row[1], row[2])
textwrite.write(string)
#if you know the second column is a float:
manipulate = float(row[1])*3
textwrite.write(manipulate)
string would then be written to 'foo.txt' as:
Ray Smith has 41645.87778 apples in his Ford V1.\n
and maniuplate would be written to 'foo.txt' as:
124937.63334

how to read a comma sep value with .txt extension into python as an array?

I am biologist and very very new to Python and before, i learnt a bit of R.
So I have a very big text file (3 GB, too big to handle in R), all values are comma seperated but the extension is .txt (I don't know if it is necessary information). what i wanted to do is to:
read it into python as an object which is equivalent of dataframe in R,
get rid of columns in the middle
reduce the size of the object
write it as txt file
take the rest to R.
If you can help me i would be very happy.
thank you

There is no real need to go into python first. Your question looks a lot like this question. The answer marked as the correct answer iteratively reads the large file, and creates a new, smaller file. Other good alternatives are using sqlite and the sqdf package, or use the ff package. This last approach works particularly well is the number of columns is small compared to the number of rows.

This will take minimal memory as it does not load the whole file at once.
import csv
with open('in.txt', 'rb') f_in, open('out.csv', 'wb') as f_out:
reader = csv.reader(f_in)
writer = csv.writer(f_out)
for row in reader:
# keep first two columns and last three columns
writer.writerow(row[:2] + row[-3:])
Note: If using Python 3 change the file modes to 'r' and 'w', respectively.

i am not familiar with r dataframe, but pandas provides helpers to read csv into pandas dataframe:
from pandas import read_csv
df = read_csv('yourfile.txt')
print df
print df['Line']
If that is not what you need you can use csv module to iterate through each line of your csv as a python list and put it into whatever data structure you want.

If you insist on using a preprocessing step, using the linux command tools is a really good and fast option. If you use Linux, these tools are already installed, under Windows you'll need to first install MinGW or Cygwin. This SO question already provides some nice pointers. In essence you use the awk tool to iteratively process the text file, creating an output text file as you go. Copying form the accepted answer of the SO question I linked:
awk -F "," '{ split ($8,array," "); sub ("\"","",array[1]); sub (NR,"",$0); sub (",","",$0); print $0 > array[1] }' file.txt
This read the file, grabs the eight column, and dumps it to a file. See the answer for more details.

Per CRAN (new features and bug fixes re: development) the new development build 3.0.0 should allow for R to use the pagefile/swap. In windows you will need to set R_MAX_MEM_SIZE to a suitably large value.