Unix uniq command

sohguanh SG

Sorry no response in IT Support and Tech Corner forum so come here post. Need help in below.

I have a problem. I need to use Unix uniq command to remove duplicating lines but with an extra twist.

Example
1 apple a
2 apple b
3 orange c
4 orange d
....

I wanna the command to consider the middle field only (i.e apple, orange) to do removing of duplicates. But it seem uniq is treating the whole line to do comparison when remove duplicates. Any idea how?

Dun ask me use M$ Excel as the command is supposed to be in a Unix script to be executed on commandline by data centre operators in a Unix environment.

TIA

Fatum SG

you mean you're actually writing a shellscript to parse a text file or something bro ? ....

if it's a shellscript you're writing, you can try parsing the middle field with regular expressions, then work on from there ...

tell me if it works ... Wink

sohguanh SG

if it's a shellscript you're writing, you can try parsing the middle field with regular expressions, then work on from there ...

Do you mean by first parsing the middle field and then do the uniq command? I wan performance and hence I tot of using juz a single uniq command to do it. If all fails I guess I must use two commands and pipe them together.

launtpc SG

I just logged in and saw your thread, but the problem is that I don't program in Unix... only VB6... sorry. Sad

Fatum SG

Originally posted by sohguanh:
[b]if it's a shellscript you're writing, you can try parsing the middle field with regular expressions, then work on from there ...

Do you mean by first parsing the middle field and then do the uniq command? I wan performance and hence I tot of using juz a single uniq command to do it. If all fails I guess I must use two commands and pipe them together.[/b]

yes, that's what I meant, if you're just parsing a simple text file then you can filter out all the unneccessary bits then pipe the output to uniq ... and frankly, i don't think there's much difference performance wise ....

I just "man uniq" .... not very helpful there .... let me know if my idea works yah ... Wink

(ahhhh ... hang on ... you may want to tinker around with -f or -s .... Razz )

Fatum SG

bro ... got the answer ...

you go uniq -f 1 [input file] [outputfile] , where 1 is the first field of each input line

i did a little experiment, cat >> test.txt a whole string of text, seperated by white space ....

and yes, it filters out the first token ....

cheers ! ... Wink

sohguanh SG

(ahhhh ... hang on ... you may want to tinker around with -f or -s .... Razz )

I just check the -f option is is always skip the FIRST fields so it means it cannot skip selected fields like say 1st and 3rd field during comparison for removing duplicates Sad

sohguanh SG

and yes, it filters out the first token ....

Correct but what I want is to filter out the first and the third token when doing comparison. So "1 apple a" should be same as "2 apple b" Mr. Green

Fatum SG

Originally posted by sohguanh:
[b]and yes, it filters out the first token ....

Correct but what I want is to filter out the first and the third token when doing comparison. So "1 apple a" should be same as "2 apple b" Mr. Green [/b]

so you just want the middle token eh ? ...

well ... a dirty hack would be to simply parse the output file again with -f 3 again lor ... Mr. Green

"uniq -f 1 [input file] | uniq -f 3 " ..... Wink

sohguanh SG

"uniq -f 1 [input file] | uniq -f 3 " .....

Errrr.... do you actually execute those commands you recommend? Do you have a linux or unix accessible to you? I tried yours and it does not work. I will try to post my working solution later assuming I do in two or more commands Mr. Green

Fatum SG

Originally posted by sohguanh:
[b]"uniq -f 1 [input file] | uniq -f 3 " .....

Errrr.... do you actually execute those commands you recommend? Do you have a linux or unix accessible to you? I tried yours and it does not work. I will try to post my working solution later assuming I do in two or more commands Mr. Green [/b]

i did it on my apple terminal and it works ....

should be no problem leh ... try again and see ? ... the above command outputs to standard out ... maybe you try piping to your desired output file and cat it and see how ? ...

Fatum SG

ah ... I see what you mean ... the output doesn't seem to be correct ...

sorry bro ... that i couldn't be much help ... but post the solution up yah ... i'd like to take a look at the final shellscript .. Wink

one-niner SG

cannot use AWK ?

Fatum SG

bro ...

i just couldn't put this down ... re-read the man again ...

the reason it's not working is cos the -f option ignores the first "x" number of fields, so -f 3 ignores the entire line, instead of ignoring the third token instead ...

hope this helps you with your solution ... good luck

sohguanh SG

the reason it's not working is cos the -f option ignores the first "x" number of fields, so -f 3 ignores the entire line, instead of ignoring the third token instead ...

I think I did mention in my earlier post I did say the FIRST "x" number of fields..

anyway me gotta it all work out but i uses 3 process of which 2 is awk maybe someone can help me tune it further to 2? The input file is called a.txt

Contents of a.txt
1 apple a
2 apple b
3 orange c
4 orange d
5 pear e
6 pear f

Result is
1 apple a
3 orange c
5 pear e

one-niner SG

hey sorry guys

given a certain string [11to200]

how can i grab out the 11 and 200 to do a comparision based on a certain value being passed in ??

I need to test conditions where a certain return codes maps to a certain number or range ...similar to one above .

sohguanh SG

given a certain string [11to200]

how can i grab out the 11 and 200 to do a comparision based on a certain value being passed in ??

You should give an example of what is the input and what you want the output to look like. Then it is clearer cuz by describing in english sentence sometimes ppl can misinterpret especially for english lousy ppl like me Mr. Green

one-niner SG

Originally posted by sohguanh:
[b]given a certain string [11to200]

how can i grab out the 11 and 200 to do a comparision based on a certain value being passed in ??

You should give an example of what is the input and what you want the output to look like. Then it is clearer cuz by describing in english sentence sometimes ppl can misinterpret especially for english lousy ppl like me Mr. Green [/b]

okay cause i got an INI file which requires me to map specfic values to user defined ranges similarly above

so lets say

criticalerror=[11to20],69
warningerror=[200to234]

given a control script that execute a sub sciript and returns 18,the control script has to map the return code 18 to a critical error which is bounded by the range [11to20]

12qwaszx SG

# New variables

set UHOMEDIR=`(cat /etc/passwd | awk -F":" '{print $6}')`
set SORTED_UHOMEDIR=`echo $UHOMEDIR | tr " " "\n" | sort -b -t" " -u`

i written something similar

the above will take the 6th field from /etc/passwd
and then sort with the unique entries
hope it helps