f



``awk '!a[$0]++' file'' and ``awk '{if(!($0 in rec)) {rec[$0]=1; print $0;}}' file ''

Hi all,

Currently, I'm reading the book on awk programming language, but meet the 
following two examples which I cann't understand so well:

awk '!a[$0]++' file

and

awk '{if(!($0 in rec)) {rec[$0]=1; print $0;}}' file

Could someone here please give me some hints or explanations on the logic 
of above codes?

Regards
-- 
..: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.
0
Hongyi
3/25/2015 1:52:54 PM
comp.unix.shell 15484 articles. 3 followers. Post Follow

9 Replies
1062 Views

Similar Articles

[PageSpeed] 15

In article <meuejm$gsf$1@aspen.stu.neva.ru>,
 Hongyi Zhao <hongyi.zhao@gmail.com> wrote:

> Hi all,
> 
> Currently, I'm reading the book on awk programming language, but meet the 
> following two examples which I cann't understand so well:

The book gives examples without explaining them itself?

> 
> awk '!a[$0]++' file

The a array will initially be empty, so a[$0] will be empty, therefore 
!a[$0] will be true, and it will print the current line. It will also 
increment a[$0], so the next time it gets to an identical line, a[$0] 
will be non-zero, and !a[$0] will be false, so it won't print the line. 
So this prints the file with duplicates removed.

> 
> and
> 
> awk '{if(!($0 in rec)) {rec[$0]=1; print $0;}}' file

This is the same as the previous, but should be more understandable. The 
recs array uses each line in the file as its keys. If there's no entry 
for the current line, it adds one and prints the line. The next time it 
gets to an identical line, !($0 in recs) will be false, so it won't do 
anything.

-- 
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
0
Barry
3/25/2015 1:01:01 AM
On Wed, 25 Mar 2015 10:48:23 -0400, Barry Margolin wrote:

>> awk '!a[$0]++' file
> 
> The a array will initially be empty, so a[$0] will be empty, therefore
> !a[$0] will be true, and it will print the current line.

Do you mean the `empty' is equivalent to zero?

If so, all of the following things are all equivalent in awk language's 
context:

undef, empty, null, and zero.

Regards
-- 
..: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.
0
Hongyi
3/25/2015 1:01:01 AM
On Wed, 25 Mar 2015 10:48:23 -0400, Barry Margolin wrote:

>> awk '!a[$0]++' file
> 
> The a array will initially be empty, so a[$0] will be empty, therefore
> !a[$0] will be true, and it will print the current line. It will also
> increment a[$0], so the next time it gets to an identical line, a[$0]
> will be non-zero, and !a[$0] will be false, so it won't print the line.
> So this prints the file with duplicates removed.

Thanks for your hints.  

Another question: is it possible to let awk print all of the information 
about the variable, array[key]=value, and bool etc. used here for each 
line?  So that I can clearly look into the inner oprations logic on my 
file applied by awk and then it will be more understandable for me.  

Regards
-- 
..: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.
0
Hongyi
3/25/2015 1:01:01 AM
On Wed, 25 Mar 2015 10:48:23 -0400, Barry Margolin wrote:

> The book gives examples without explaining them itself?

I read the book, but learning some more examples from out of the book for 
relevant chapters ;-(

Regards
-- 
..: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.
0
Hongyi
3/25/2015 10:51:12 PM
On Wed, 25 Mar 2015 23:32:57 +0000 (UTC), Hongyi Zhao
<hongyi.zhao@gmail.com> wrote:

>On Wed, 25 Mar 2015 10:48:23 -0400, Barry Margolin wrote:
>
>>> awk '!a[$0]++' file
>> 
>> The a array will initially be empty, so a[$0] will be empty, therefore
>> !a[$0] will be true, and it will print the current line.
>
>Do you mean the `empty' is equivalent to zero?

An empty array is an array witout elements.
In awk, if a variable is referenced, but it has not been assigned
a variable yet, it returns 0 (zero) in a numerical context, and ""
(empty string) in a string context. That also applies to array
elements. 
The extra feature for array elements is that you can test for
existence with: if (key in array) ...

That is all in the manual.


>If so, all of the following things are all equivalent in awk language's 
>context:
>
>undef, empty, null, and zero.

More or less, yes.

Regards
-- 
Kees Nuyt
0
Kees
3/26/2015 10:41:45 AM
On Wed, 25 Mar 2015 23:47:25 +0000 (UTC), Hongyi Zhao
<hongyi.zhao@gmail.com> wrote:

>Another question: is it possible to let awk print all of the information 
>about the variable, array[key]=value, and bool etc. used here for each 
>line?

Sure.

for (idx in array) printf "array[%s] = %s\n",idx,array[idx]

> So that I can clearly look into the inner oprations logic on my 
>file applied by awk and then it will be more understandable for me.  

http://www.gnu.org/software/gawk/manual/html_node/Variable-Typing.html#Variable-Typing


Regards
-- 
Kees Nuyt
0
Kees
3/26/2015 10:50:19 AM
Grrr, typoes

An empty array is an array without elements.
In awk, if a variable is referenced, but it has not been assigned
a value yet, it returns 0 (zero) in a numerical context, and ""
(empty string) in a string context. That also applies to array
elements. 
The extra feature for array elements is that you can test for
existence with: if (key in array) ...

-- 
Kees Nuyt
0
Kees
3/26/2015 10:54:54 AM
On 3/25/2015 8:52 AM, Hongyi Zhao wrote:
> Hi all,
>
> Currently, I'm reading the book on awk programming language, but meet the
> following two examples which I cann't understand so well:
>
> awk '!a[$0]++' file
>
> and
>
> awk '{if(!($0 in rec)) {rec[$0]=1; print $0;}}' file

No. There is no "book on awk programming language" that would show that second 
script as an example of how to use awk.

>
> Could someone here please give me some hints or explanations on the logic
> of above codes?

Actually do get and read the book on awk - Effective Awk Programming by Arnold 
Robbins. All other books on awk are out of date or otherwise not as useful.

    Ed.

0
Ed
3/28/2015 3:04:53 PM
Am 28.03.2015 um 16:04 schrieb Ed Morton:
>
> Actually do get and read the book on awk - Effective Awk Programming by
> Arnold Robbins. All other books on awk are out of date or otherwise not
> as useful.

Also the original book from A. K. and W., while old and expensive, is
very good! The basic chapters are terse and well written. For people
who have problems getting the basics it's (in this specific respect)
probably even better than the more thorough and modern book(s).

Janis

0
Janis
3/28/2015 8:04:12 PM
Reply:

Similar Artilces:

'is not' or '!='
A newbie question to you; what is the difference between statements like: if x is not None: and if x != None: Without any context, which one should be preferred? IMHO, the latter is more readable. On 2014-08-18 21:35, ElChino wrote: > A newbie question to you; what is the difference between statements > like: > if x is not None: > and > if x != None: > > Without any context, which one should be preferred? > IMHO, the latter is more readable. > "x == y" tells you whether x and y refer to objects that are equal. "x is y&qu...

'^=' and '~='?
Hello, What is the difference between '^=' and '~='? Thanks, Duckhye ...

{ '0':'c->c->a' ,'1':'a->b->a' .........}
Hi, have anybody a hint , how i get a dict from non unique id's and their different related values. Thanks for advance Chris ###random data # a=range(10)*3 def seqelem(): i=random.randint(0,2) elem=['a','b','c'][i] return elem s=[seqelem() for t in range(30)] print zip(a,s) ## favored result: { '0':'c->c->a' ,'1':'a->b->a' .........} Hi Chris, I may have time to look at the rest of your code later. For now I just want to comment on one line: On Nov 7, 12:24=A0pm, chris <o...

'''''''''''''The Running Update/Append Queries Using VBA code Ordeal''''''''''''''
Hello fellow programmers, I am trying to run an append/update query from code, a command button on a form initiates the queries. the format i am using is; _____________________________________________________ SELECT "criteria" FROM "criteria" WHERE "criteria" UPDATE/APPEND "field selections" RecordSource "qryExample" = above text strings" _______________________________________________________________________ When i am running a SELECT query in this manner it works fine with no problems, and accepts the values of specified linked for...

string search for '0' and/or '1'
I need to process a file which has lines of just '1's & '0's, or something else. I want to process all the lines that are only '0' & '1's and ignore the rest. So far I have used: while { [gets $f1 inline1] >= 0} { # scan line for non-binary characters, skip line if present: if { ([string index $inline1 0] == "0") || ([string index $inline1 0] == "1") } { ..... ...... But this only tests the first char (which does work, but I think there might be a better way) Is there a way to check the whole of $inline for chars other tha...

Dumb if ('1' == '0') question
Hi there, Please be gentle - I'm never used Javascript, but I need to reverse-engineer my router, which my ISP (whom I bought it from) have helpfully locked to their service. I managed to find a hidden webpage on the router which allows one to enter an unlock code, and if I enter (say) 999 in this box it takes me to a webpage with the address <http://192.168.1.200/brdg.cgi?brdg=999>. That page helpfully shows a message announcing "Your router is still locked." It was my intention to brute-force the unlock code, by writing a bash-script to download every URL from <http...

'''''''''''''The Running Update/Append Queries Using VBA code Ordeal'''''''''''''' #2
Hi, Thanks for ur help there HJ. I know how to do the tasks you specified there. I would like for the update query to use field values from some of the fields on the form (frmInvoices) such as InvoiceNumber, DateFrom, DateTo. My problem is that an append/update query can't find the values in the open Form (frmInvoices) when I specify them as; [Forms]![frmInvoices]![InvoiceNumber] a select query has no problem finding the field values on a form. please help. Aaron Hi Aaron, Could you post the entire code that you are having trouble with? Now it is not possible to see what goes wron...

if str_mo not in ('','.') and str_da not in ('','.') and str_yy not in ('','.') Any shorter ?
Hi, there. =20 I'm just curious if it ever dawned on anybody how to abbreviate this line : if str_mo not in ('','.') and str_da not in ('','.') and str_yy not in ('','.')=20 =20 Igor Kurbeko Clinical Programmer Analyst 678 336 4328 ikurbeko@atherogenics.com =20 no brain no pain =20 how about: if not (str_mo in ('','.') or str_da in ('','.') or str_yy in ('','.')) OR if not (missing(str_mo) or missing(str_da) or missing(str_yy)) Eric On 22 Oct 03 21:13:37 GMT, ikurbeko@ATHER...

A function with 'and' , 'not' , 'null' , 'car' and 'cdr'
What's this ? (defun enigma (x) (and (not (null x)) (or (null (car x)) (enigma (cdr x))))) "I suppose I should learn Lisp, but it seems so foreign." - Paul Graham, Nov 1983 On Wed, Oct 07 2015, CAI GENGYANG wrote: > What's this ? > > > (defun enigma (x) > (and (not (null x)) > (or (null (car x)) > (enigma (cdr x))))) Bad taste? It returns T if the list X contains nil as an element. It would be clearer to write (some #'null x). Helmut CAI GENGYANG ...

'11' + '1' is '111'?
'11' + '1' == '111' is well known. but it suprises me '11'+'1' IS '111'. Why? Obviously they are two differnt object. Is this special feature of imutable object? On Thu, Oct 29, 2009 at 5:43 PM, metal <metal29a@gmail.com> wrote: > '11' + '1' == '111' is well known. > > but it suprises me '11'+'1' IS '111'. > > Why? Obviously they are two differnt object. > > Is this special feature of imutable object? It's an implementation detail used to optimize performance. CPy...

Re: if str_mo not in ('','.') and str_da not in ('','.') and str_yy not in ('','.') Any shorter ?
OR you could use ARRAY data new; set old; array igor $ (*) str_mo str_da str_yr; do over igor; if igor ~in (' ','.') then do; end; run; Prasad Ravi Igor Kurbeko <ikurbeko@ATHEROGENIC To: SAS-L@LISTSERV.UGA.EDU S.COM> cc: Sent by: "SAS(r) Subject: if str_mo not in ('','.') and str_da not in ('','.') and str_yy ...

'1' or '0' when I/O pin is pulled up
Hi all, I am sorry if this problem has been answered but I cannot find much help from the archive. I am trying to write and read registers from a CMOS sensor via I2C. The SCLK and SDATA pins are pulled by 3.3V and are connected to two I/O pins on Spartan3 FPGA. When the FPGA is reading from SDATA, would it be always high ('1' bit instead of '0') as it's pulled up?? Thanks! Nick On Wed, 13 Feb 2008 00:24:54 +0800, Nick <tklau@cuhk.edu.hk> wrote: >I am trying to write and read registers from a CMOS sensor via I2C. The >SCLK and SDATA pins are pulled by...

error: expected '=', ',', ';', 'asm' or '__attrib
Hi I'm trying to compile an ADC Driver & come acrosss the following error. I've no experience writing drivers before, and hence have no clue how to fix it. Hope someone out there has encountered the problem & suggesst a fix for the same. The Error is I get is : qadc.c: At top level: qadc.c:97: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'qadc_read' make: *** [qadc.o] Error 1 [root@localhost qadc]# ########################################################################### ADC Driver Code ##...

error: expected '=', ',', ';', 'asm' or '__attrib
Hi I'm trying to compile an ADC Driver & come acrosss the following error. I've no experience writing drivers before, and hence have no clue how to fix it. Hope someone out there has encountered the problem & suggesst a fix for the same. The Error is I get is : qadc.c: At top level: qadc.c:97: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'qadc_read' make: *** [qadc.o] Error 1 [root@localhost qadc]# ########################################################################### ADC Driver Code ####################...