HELP: Computer Programmers (i think)

[bushwakka]

I have a txt file which contains the records of pressure-time data. There is an insane amount of records or rows in this file.

The problem is that I have to use this data in a program to analyse something. However, at several points in the txt file, records or rows are repeated (duplicated) exactly. This gives a time continuity problem in the program I'm using so I have to delete the duplicate rows. There is an insane amount of duplicate rows and its virtually impossible to do this manually. I need a method to first parse the txt file records into excel I think and then do some kinda program/macro to find and delete the duplicates?

Help please?

oh, the number of rows is about 893,000

[ismithx]

do you want me to make such a program for you?

[RBphoto]

I will do it... 10 cents per row ...

But seriously

What you can do is paste into word. In the find dialog, do a find and replace. Then.. replace all.

Will be done in a few mins.

[RBphoto]

Mind me asking what application did the logging and what you are trying to read the text log with?

[ismithx]

I will do it... 10 cents per row ...

But seriously

What you can do is paste into word. In the find dialog, do a find and replace. Then.. replace all.

Will be done in a few mins.

hehehehe you want word to crash?

as i say bushwakka i could organize and make a program for that.... for a small price ($100-$200)

[dry]

http://unxutils.sourceforge.net/
you want uniq.exe

c:\> uniq.exe infile.txt outfile.txt

[bushwakka]

Mind me asking what application did the logging and what you are trying to read the text log with?

dunno what app made the log, im importing into pansystem to do the well test analysis

You cant use find and replace in word cuz the duplicates are not unique.....it looks like this:

01/18/2010 10:11:31 4279.786133
01/18/2010 10:11:31 4279.786133
01/18/2010 10:11:36 4279.787598
01/18/2010 10:11:36 4279.787598
01/18/2010 10:11:41 4279.788086
01/18/2010 10:11:41 4279.788086

a friend pointed out this feature to me in excel
http://office.microsoft.com/en-us/excel ... 34626.aspx

im gonna parse the text file into a table in word by using the convert text to table option (hope computer doesnt frig up) and then paste that table into excel and try the above

...crosses fingers

[RBphoto]

32MB is the maximum word file size... that is all I know of the limitations. Anyway, I used stuff like Jpad pro and 1st page to do this in years gone by. Don't know what is the best editor nowdays that can do this find and replace.

[RBphoto]

Hmmm, looking at this, if you can import to a database (like access or mysql), you can use wildcard searches and do more efficient filters. Cool thing is, once done with a database filter, you can run a report and have your stuff formated any which way you need.

[bushwakka]

ok....so i encountered a problem......i cant make a table in word that is so long.....so i have to find another way to parse the records from the text file into a table......suggestions?

[dry]

dude, try the uniq command from the command prompt.

it takes an arbitrary text file and spits it back out with the duplicates removed. exactly what you want.

[RBphoto]

1st page 2000? is a free download, and I think you can get search and replace capability. Ditto with Jpad pro. Try them out (nothing to loose I guess)

[Bizzare]

Are the duplicates doubled like in the example above, or are there some areas where it triples etc?

[given]

MSExcel go help yuh out use data filters to remove dups


only prob is to get it into excel i sure it have something out there man good luck !

[bushwakka]

Are the duplicates doubled like in the example above, or are there some areas where it triples etc?

only doubles....i gonna check out crossdrilled programs

dude, try the uniq command from the command prompt.

it takes an arbitrary text file and spits it back out with the duplicates removed. exactly what you want.

more info please on this uniq thing.....i seeing sumn to do with linux and all that jazz and i not sure i understand it

[benko]

dude, try the uniq command from the command prompt.

it takes an arbitrary text file and spits it back out with the duplicates removed. exactly what you want.

^it requires that the file be sorted beforehand (it will only remove duplicates on adjacent lines), it is a windows commandline app that was ported from unix

I would suggest importing the file into any database app or even simply use excel if your file is not sorted or you need to work through a few sanitization steps

[psykomorf]

If u want to use excel, try importing the text file into excel
http://office.microsoft.com/en-us/excel-help/import-or-export-text-txt-or-csv-files-HP010099725.aspx

then delete duplicate rows
http://office.microsoft.com/en-us/excel-help/delete-duplicate-rows-from-a-list-in-excel-HA001034626.aspx

[Smokey]

*cough* basic excel *cough*

[bushwakka]

wow ok....thanks psyko....cant believe the answer to importing in excel was so ez all along...did the import, now trying the filters....taking a lil while

[Sky]

*in French Accent* 200 Years later..

[TK!]

excel as they said. was gonna say how, but psyk... said it

[bushwakka]

ok....excel didnt work out, at least on my comp.....893,000 + rows of data seems too much for it to sort, it just locks up...i left it for like 4 hrs

my friend said he'll do some magic with a mysql database or sumn so

[Bizzare]

Copy and paste like 25 of the duplicated rows here if you can/ if you still need help.

[FullStop]

um, that is an easy piece of code to write, how big is the text file? in terms of MB? email me the text file, iwill try and write the code to do it tonight... ghostbustersss at gmail dot com, hit me a pm when you send it.

[Bizzare]

ghostbusters, expects lots and LOTS of spam!!
Putting ur email on a public forum....great idea

[Bizzare]

But yea, Sounds like a simple piece of code. Could write a code an allow you to parse the file through yuh browser and remove d duplicates.

[bushwakka]

ok...i uploaded the rar file to rapidshare
here is the link: https://rapidshare.com/files/2971643094/pressures.rar

u guys can give it a go

ok, a friend helped me a bit, he did some magic with some database app and he got rid of the duplicates

some of the readings are now out of order though and they have to be put in order to ensure time continuity otherwise my analysis program gives an error.....here's an update to the original file

https://rapidshare.com/files/3549555402 ... sorted.zip

[Bizzare]

So I write a script and remove duplicates. But I not clear on what duplicates to remove. D ones with exact date, time and pressure? Or only the ones with exact pressures? Currently it remove the duplicates with exact date, time and pressure like in your example above.

[psykomorf]

not sure if this is what u looking for
https://rapidshare.com/files/4097995993/pressures.zip

[FullStop]

my bad there bizzare, i a bit out of it, am currently piped up on muscle relaxer...look slike bushwakka get through, if any further assistance is needed, i around...