Reading and doing calculation from .dat file in python
Let's say I have a .dat file that has millions of lines of rows and 12 columns in total. I need to have some calculations, for that purpose I need to divide columns number 2,3, and 4 with column number 1. So, Before loading the .dat file, Whether I need to delete all other unwanted columns?
Take data.dat as an example for .dat file
Since I'm new to python, It would be appreciated if the instruction on open, read and calculation is explained.
I have also added the code, have a look at it:
from sys import argv
import pandas as pd
script, filename = argv
txt = open(filename)
print "Here's your file %r:" % filename
print txt.read()
def your_func(row):
return row['x-momentum'] / row['mass']
columns_to_keep = ['mass', 'x-momentum']
dataframe = pd.read_csv('~/Pictures', delimiter="," , usecols=columns_to_keep)
dataframe['new_column'] = dataframe.apply(your_func, axis=1)
The error which I get:
Traceback (most recent call last):
File "flash.py", line 18, in
dataframe = pd.read_csv('~/Pictures', delimiter="," , usecols=columns_to_keep)
File "/home/trina/anaconda2/lib/python2.7/site-packages/pandas/io/parsers.py", line 529, in parser_f
return _read(filepath_or_buffer, kwds)
File "/home/trina/anaconda2/lib/python2.7/site-packages/pandas/io/parsers.py", line 295, in _read
parser = TextFileReader(filepath_or_buffer, **kwds)
File "/home/trina/anaconda2/lib/python2.7/site-packages/pandas/io/parsers.py", line 612, in __init__
self._make_engine(self.engine)
File "/home/trina/anaconda2/lib/python2.7/site-packages/pandas/io/parsers.py", line 747, in _make_engine
self._engine = CParserWrapper(self.f, **self.options)
File "/home/trina/anaconda2/lib/python2.7/site-packages/pandas/io/parsers.py", line 1119, in __init__
self._reader = _parser.TextReader(src, **kwds)
File "pandas/parser.pyx", line 518, in pandas.parser.TextReader.__cinit__ (pandas/parser.c:5030)
ValueError: No columns to parse from file
Are you facing pandas read dat file error while coding. If yes, then you should had a glance at your "flash.dat" file, Before processing it, you actually need to do some cleanup.
The below code will help you to convert it to a CSV
import csv
# read flash.dat to a list of lists
datContent = [i.strip().split() for i in open("./flash.dat").readlines()]
# write it as a new CSV file
with open("./flash.csv", "wb") as f:
writer = csv.writer(f)
writer.writerows(datContent)
Now, To compute a new column we will use Pandas
import pandas as pd
def your_func(row):
return row['x-momentum'] / row['mass']
columns_to_keep = ['#time', 'x-momentum', 'mass']
dataframe = pd.read_csv("./flash.csv", usecols=columns_to_keep)
dataframe['new_column'] = dataframe.apply(your_func, axis=1)
print dataframe