f



gawk 4.1.3 gensub() warning?

Anyone else seeing this warning with gawk 4.1.3:

$ echo 'a' | gawk '{print gensub(/a/,"b","")}'
gawk: cmd. line:1: (FILENAME=- FNR=1) warning: gensub: third argument `' treated 
as 1
b

Any workaround other than changing the 3rd arg to 1 in all of my scripts?

	Ed.
0
Ed
9/3/2015 3:51:29 AM
comp.lang.awk 3450 articles. 0 followers. Post Follow

10 Replies
1393 Views

Similar Articles

[PageSpeed] 21

On 2015-09-03 05:51 Ed Morton wrote:

> Anyone else seeing this warning with gawk 4.1.3:
>
> $ echo 'a' | gawk '{print gensub(/a/,"b","")}'
> gawk: cmd. line:1: (FILENAME=- FNR=1) warning: gensub: third argument `'
> treated as 1
> b
>

Same here.

This is documented behavior (but not quite, to be honest):

"If how is a string beginning with ‘g’ or ‘G’, then it replaces all 
matches of regexp with replacement. Otherwise, how is treated as a 
number that indicates which match of regexp to replace. [...] If the how 
argument is a string that does not begin with ‘g’ or ‘G’, or if it is a 
number that is less than or equal to zero, only one substitution is 
performed. If how is zero, gawk issues a warning message."

in addition: "the empty string [...] is zero if converted to a number"


The only thing departing from documentation is that in addition to only 
making one substitution the warning also is given when using a string as 
the "how" argument that has a value other than 0 if converted to a 
number e.g. "3aif" (i.e. starting with a digit). So maybe the docs might 
be changed to "... only one substitution is performed, and gawk issues a 
warning message."

This behavior is different from older versions (e.g. 3.1.6) where the 
warning was only given when the how argument was explicitly 0 or "0".

> Any workaround other than changing the 3rd arg to 1 in all of my scripts?
>
>      Ed.

Using an older version? (Just joking ;-)


jf

0
Josef
9/3/2015 7:56:43 AM
On 9/3/2015 2:56 AM, Josef Frank wrote:
> On 2015-09-03 05:51 Ed Morton wrote:
>
>> Anyone else seeing this warning with gawk 4.1.3:
>>
>> $ echo 'a' | gawk '{print gensub(/a/,"b","")}'
>> gawk: cmd. line:1: (FILENAME=- FNR=1) warning: gensub: third argument `'
>> treated as 1
>> b
>>
>
> Same here.
>
> This is documented behavior (but not quite, to be honest):
>
> "If how is a string beginning with ‘g’ or ‘G’, then it replaces all matches of
> regexp with replacement. Otherwise, how is treated as a number that indicates
> which match of regexp to replace. [...] If the how argument is a string that
> does not begin with ‘g’ or ‘G’, or if it is a number that is less than or equal
> to zero, only one substitution is performed. If how is zero, gawk issues a
> warning message."
>
> in addition: "the empty string [...] is zero if converted to a number"
>
>
> The only thing departing from documentation is that in addition to only making
> one substitution the warning also is given when using a string as the "how"
> argument that has a value other than 0 if converted to a number e.g. "3aif"
> (i.e. starting with a digit). So maybe the docs might be changed to "... only
> one substitution is performed, and gawk issues a warning message."
>
> This behavior is different from older versions (e.g. 3.1.6) where the warning
> was only given when the how argument was explicitly 0 or "0".
>
>> Any workaround other than changing the 3rd arg to 1 in all of my scripts?
>>
>>      Ed.
>
> Using an older version? (Just joking ;-)
>
>
> jf
>

Thanks for the info. This new warning is like if sed suddenly started issuing 
warnings if you wrote `s/foo/bar/` instead of `s/foo/bar/1`. Since others are 
seeing it too I'll follow up with bug-awk@gnu.org.

       Ed.
0
Ed
9/3/2015 11:49:39 AM
In article <ms9c19$hus$1@dont-email.me>,
Ed Morton  <mortonspam@gmail.com> wrote:
....
>Thanks for the info. This new warning is like if sed suddenly started issuing 
>warnings if you wrote `s/foo/bar/` instead of `s/foo/bar/1`. Since others are 
>seeing it too I'll follow up with bug-awk@gnu.org.
>
>       Ed.

It's not a bug.  The documentation was always clear that the acceptable
values were either 1 or G/g.  You were always in violation of that.  Your
sed analogy is completely bogus (as expected).

BTW, I lobbied for (and got) this better error reporting into gensub().

-- 
Mike Huckabee has yet to consciously uncouple from Josh Duggar.

0
gazelle
9/3/2015 12:01:07 PM
In article <ms9cq3$6td$1@news.xmission.com>,
Kenny McCormack <gazelle@shell.xmission.com> wrote:
....
>It's not a bug.  The documentation was always clear that the acceptable
>values were either 1 or G/g.

Correction: This should, of course, be "a positive integer" not (only) "1".
Also, I think any string that begins with either "G" or "g" is OK as well.

-- 
"There's no chance that the iPhone is going to get any significant market share. No chance." - Steve Ballmer
0
gazelle
9/3/2015 3:13:56 PM
On 2015-09-03, Kenny McCormack <gazelle@shell.xmission.com> wrote:
> In article <ms9cq3$6td$1@news.xmission.com>,
> Kenny McCormack <gazelle@shell.xmission.com> wrote:
> ...
>>It's not a bug.  The documentation was always clear that the acceptable
>>values were either 1 or G/g.
>
> Correction: This should, of course, be "a positive integer" not (only) "1".
> Also, I think any string that begins with either "G" or "g" is OK as well.

Indeed, the documentation says that "if how is a string beginning with ‘g’ or
‘G’ (short for “global”), then replace all matches of regexp with replacement".

That doesn't mean it's a good idea to put something after that 'g' or 'G',
because that could be an area of future extension.

(If there is no such extension now, the documentation should just say, 'the
string "g" or "G"'.  It's a mistake to give a seeming assurance that trailing
characters are ignored, if that assurance might be taken away.)
0
Kaz
9/3/2015 5:55:39 PM
On 9/3/2015 7:01 AM, Kenny McCormack wrote:
> In article <ms9c19$hus$1@dont-email.me>,
> Ed Morton  <mortonspam@gmail.com> wrote:
> ...
>> Thanks for the info. This new warning is like if sed suddenly started issuing
>> warnings if you wrote `s/foo/bar/` instead of `s/foo/bar/1`. Since others are
>> seeing it too I'll follow up with bug-awk@gnu.org.
>>
>>        Ed.
>
> It's not a bug.  The documentation was always clear that the acceptable
> values were either 1 or G/g.  You were always in violation of that.  Your
> sed analogy is completely bogus (as expected).
>
> BTW, I lobbied for (and got) this better error reporting into gensub().
>

Great work Kenny, a valuable contribution to the community. Maybe you now could 
lobby for a warning message when the 2nd arg to substr() is out of range:

$ awk 'BEGIN{print substr("foobar",-127,3)}'
foo

$ awk 'BEGIN{print substr("foobar","",3)}'
foo

since that would create even more unnecessary work for even more people (though 
in the substr() case it might actually be finding a problem, unlike the gensub() 
case).

	Ed.
0
Ed
9/4/2015 2:15:55 AM
Am 04.09.2015 um 05:15 schrieb Ed Morton:
> On 9/3/2015 7:01 AM, Kenny McCormack wrote:
>> In article <ms9c19$hus$1@dont-email.me>,
>> Ed Morton  <mortonspam@gmail.com> wrote:
>> ...
>>> Thanks for the info. This new warning is like if sed suddenly started
>>> issuing
>>> warnings if you wrote `s/foo/bar/` instead of `s/foo/bar/1`. Since
>>> others are
>>> seeing it too I'll follow up with bug-awk@gnu.org.
>>>
>>>        Ed.
>>
>> It's not a bug.  The documentation was always clear that the acceptable
>> values were either 1 or G/g.  You were always in violation of that.  Your
>> sed analogy is completely bogus (as expected).
>>
>> BTW, I lobbied for (and got) this better error reporting into gensub().
>>
>
> Great work Kenny, a valuable contribution to the community. Maybe you
> now could lobby for a warning message when the 2nd arg to substr() is
> out of range:
>
> $ awk 'BEGIN{print substr("foobar",-127,3)}'
> foo
>
> $ awk 'BEGIN{print substr("foobar","",3)}'
> foo
>
> since that would create even more unnecessary work for even more people
> (though in the substr() case it might actually be finding a problem,
> unlike the gensub() case).

Curious; what was your primary intention to use an empty string
in gensub and not one of the documented/defined parameter values?

Your substr statement made me curious, and I noticed that it's
not directly possible to use negative indices to implement a
sliding substring window across a string:

$ awk 'BEGIN{s="ABCD"; for (i=-2;i<=length(s);i++) print substr(s,i,3)}'
ABC
ABC
ABC
ABC
BCD
CD
D

At the end of the string the data is trimmed, but not at the front
of the string. So, yes, a warning - assuming unchanged behaviour -
might even be appropriate.

Janis

>
>      Ed.

0
Janis
9/4/2015 8:28:24 AM
On 9/4/2015 3:28 AM, Janis Papanagnou wrote:
> Am 04.09.2015 um 05:15 schrieb Ed Morton:
>> On 9/3/2015 7:01 AM, Kenny McCormack wrote:
>>> In article <ms9c19$hus$1@dont-email.me>,
>>> Ed Morton  <mortonspam@gmail.com> wrote:
>>> ...
>>>> Thanks for the info. This new warning is like if sed suddenly started
>>>> issuing
>>>> warnings if you wrote `s/foo/bar/` instead of `s/foo/bar/1`. Since
>>>> others are
>>>> seeing it too I'll follow up with bug-awk@gnu.org.
>>>>
>>>>        Ed.
>>>
>>> It's not a bug.  The documentation was always clear that the acceptable
>>> values were either 1 or G/g.  You were always in violation of that.  Your
>>> sed analogy is completely bogus (as expected).
>>>
>>> BTW, I lobbied for (and got) this better error reporting into gensub().
>>>
>>
>> Great work Kenny, a valuable contribution to the community. Maybe you
>> now could lobby for a warning message when the 2nd arg to substr() is
>> out of range:
>>
>> $ awk 'BEGIN{print substr("foobar",-127,3)}'
>> foo
>>
>> $ awk 'BEGIN{print substr("foobar","",3)}'
>> foo
>>
>> since that would create even more unnecessary work for even more people
>> (though in the substr() case it might actually be finding a problem,
>> unlike the gensub() case).
>
> Curious; what was your primary intention to use an empty string
> in gensub and not one of the documented/defined parameter values?

I don't know where I got the idea but I suspect I assumed gensub() was the same 
as sed in that that arg should be g/G for global, or a number for a specific 
occurrence or empty for the first occurrence. I may even have googled examples 
and seen it used that way, e.g. http://pbraun.nethence.com/unix/lang/awk.html or 
https://www.bignerdranch.com/blog/a-crash-course-in-awk/.

Whatever got me started, I've been doing it since I first started using gawk and 
since printing a warning for the "how" being empty is new functionality, I 
suspect the gawk man page didn't contain the statement:

     If how is zero, gawk issues a warning message.

until recently and without that I would have had no reason to think I HAD to put 
a 1 in there since there's no example given in the manual of altering just the 
first match and the rest of the info says:

     how is treated as a number indicating which match of regexp to replace

and

     If the how argument ... is a number that is less than or equal to zero, 
only one substitution is performed.

so while glancing through the manual none of that would have lead me to think I 
was wrong in just using "".

Even if the statement "If how is zero, gawk issues a warning message." had 
always been present but that functionality not implemented I could easily have 
seen myself thinking it made sense to report gensub(/o/,"x",0,"foobar") since 
that'd also be an error in sed:

$ echo 'foobar' | sed 's/o/x/0'
sed: -e expression #1, char 7: number option to `s' command may not be zero

but not relating that to gensub(/o/,"x","","foobar") being a problem since it's 
not a problem in sed to not specify that qualifier.

It may all come back to me thinking that fundamentally gensub() is giving us 
sed-like functionality in awk.

> Your substr statement made me curious, and I noticed that it's
> not directly possible to use negative indices to implement a
> sliding substring window across a string:

Not entirely sure what that means but I threw a negative value in there just to 
show it's possible, the buggy code I typically do come across is:

$ awk 'BEGIN{s="ABCD"; for (i=length(s);i>=0;i--) print substr(s,i,1)}'
D
C
B
A
A

It's easy to see the bug with a small string and printing the values each 
iteration but of course that's not always what's happening in the loop.

Regards,

	Ed.

>
> $ awk 'BEGIN{s="ABCD"; for (i=-2;i<=length(s);i++) print substr(s,i,3)}'
> ABC
> ABC
> ABC
> ABC
> BCD
> CD
> D
>
> At the end of the string the data is trimmed, but not at the front
> of the string. So, yes, a warning - assuming unchanged behaviour -
> might even be appropriate.
>
> Janis
>
>>
>>      Ed.
>

0
Ed
9/4/2015 12:42:02 PM
Am 04.09.2015 um 15:42 schrieb Ed Morton:
> On 9/4/2015 3:28 AM, Janis Papanagnou wrote:
>>
>> Curious; what was your primary intention to use an empty string
>> in gensub and not one of the documented/defined parameter values?
>
> I don't know where I got the idea but I suspect I assumed gensub() was
> the same as sed in that that arg should be g/G for global, or a number
> for a specific occurrence or empty for the first occurrence. [...]
[...]
> It may all come back to me thinking that fundamentally gensub() is
> giving us sed-like functionality in awk.

I don't think that comparisons of awk with sed are helpful here. But
would anything speak against allowing "" as third parameter in gawk's
gensub() with the meaning of "g" (which IMO makes more sense than
being interpreted as 1)? Well, okay; it would then behave differently
than before, but if that value was undefined anyway... - I'm not sure.

It would also be useful to have an extension to support replacements
counted from the rear of the string, say,  gensub(/a/,"b",-1) . It's
cumbersome to work around it.

>
>> Your substr statement made me curious, and I noticed that it's
>> not directly possible to use negative indices to implement a
>> sliding substring window across a string:
>
> Not entirely sure what that means [...]

I meant to replace (or iterate over) - instead of the output quoted
below - a sequence: A, AB, ABC, BCD, CD, D.

Graphically:  "SOME STRING"
Substring:  |---|
              |---|
                    ...
                         |---|
                          |---|

The "|---|" is a "window" (substring) sliding across the data.

Reading through iterations like substr(var,-2,3) will also always start
from 1 if negative values are provided for the start index, so that an
iteration starting the loop in the negative range will provide strange
looking results. This is contrary to reading past the end of a string.

I presume that the long existing substr() behaviour will make that and
the corresponding gensub() behaviour immutable.

Janis

>
>>
>> $ awk 'BEGIN{s="ABCD"; for (i=-2;i<=length(s);i++) print substr(s,i,3)}'
>> ABC
>> ABC
>> ABC
>> ABC
>> BCD
>> CD
>> D
>>
>> At the end of the string the data is trimmed, but not at the front
>> of the string. So, yes, a warning - assuming unchanged behaviour -
>> might even be appropriate.
>>
>> Janis
>>
>>>
>>>      Ed.
>>
>

0
Janis
9/4/2015 1:18:59 PM
In article <msaupi$jas$1@dont-email.me>,
Ed Morton  <mortonspam@gmail.com> wrote:
>On 9/3/2015 7:01 AM, Kenny McCormack wrote:
>> In article <ms9c19$hus$1@dont-email.me>,
>> Ed Morton  <mortonspam@gmail.com> wrote:
>> ...
>>> Thanks for the info. This new warning is like if sed suddenly started issuing
>>> warnings if you wrote `s/foo/bar/` instead of `s/foo/bar/1`. Since others are
>>> seeing it too I'll follow up with bug-awk@gnu.org.
>>>
>>>        Ed.
>>
>> It's not a bug.  The documentation was always clear that the acceptable
>> values were either 1 or G/g.  You were always in violation of that.  Your
>> sed analogy is completely bogus (as expected).
>>
>> BTW, I lobbied for (and got) this better error reporting into gensub().
>>
>
>Great work Kenny, a valuable contribution to the community. Maybe you now could 
>lobby for a warning message when the 2nd arg to substr() is out of range.

Incidentally, just for the benefit of those whose "Google-foo" may not be
quite up-to-snuff, here's the post where I noted that this bug had been
fixed.  Note that the script in question, the very useful "comma" function,
was, in fact, written by Ed.  So, we have Ed to thank for the finding and
fixing of this little bug.  As George Carlin says in one of his routines,
"Thanks, Ed!"

--- Cut Here ---
In article <m51vh4$bcd$2@news.xmission.com>,
Kenny McCormack <gazelle@shell.xmission.com> wrote:
>Some things I noticed about gensub()'s "how" arg:
>
>1) In 3.1.4, it seems like the "number other than 1" functionality is not
>implemented.  I.e., it looks like if the value is anything other than a
>string matching /[Gg].*/, it is taken as being "1".  In 3.1.8, it works
>correctly, so I assume it got fixed betwen those two versions.  Can anyone
>confirm this for me?  Note that although I have binaries (executables) for
>these old versions, I don't have man pages for them, so I can't check for
>myself.  Actually, what I'm really asking is "Does the man page for 3.1.4
>document it as doing what it should be doing or does it document what it
>actually was doing (in that version) ?"
>
>2) In the current version, if you pass "0" for "how", you get a warning,
>but passing a negative value (or any other "garbage" string) generates no
>warning (but all such values are treated as if they were 1):
>
>$ enhance gawk4 '{ print gensub($1,$2,$3,$4) }'
>abc DEF 1 abcabcdefghi
>DEFabcdefghi
>abc DEF 2 abcabcdefghi
>abcDEFdefghi
>abc DEF 0 abcabcdefghi
>gawk4: cmd. line:1: (FILENAME=- FNR=3) warning: gensub: third argument of 0 treated as 1
>DEFabcdefghi
>abc DEF -1 abcabcdefghi
>DEFabcdefghi
>abc DEF garbage abcabcdefghi
>DEFDEFdefghi
>abc DEF rubbish abcabcdefghi
>DEFabcdefghi
>$

An update:

This seems to be fixed in 4.1.3.  It complains about anything unexpected
(i.e., other than 1 or "G" or "g") in the third arg.  See below:

('gawk4' is version 4.1.3, 'gawk41' is version 4.1.1)
 % gawk4 'BEGIN { print gensub("abc","DEF",-1,"abcdefghikl")}'
gawk4: cmd. line:1: warning: gensub: third argument -1 treated as 1
DEFdefghikl
 % gawk41 'BEGIN { print gensub("abc","DEF",-1,"abcdefghikl")}'
DEFdefghikl
 %

Yey!  It seems complaints here do get noticed (sometimes...)

Also note, I have the following script installed on various of my systems
to "comma-ize" numbers.  I copied it from this board some time back:

--- Cut Here ---
function comma(num) {
    if (num < 0)
        return "-" comma(-num)
    while (num != (num=gensub(/([0-9])([0-9][0-9][0-9])($|[,.])/,"\\1,\\2\\3","",num)));
    return num
    }
--- Cut Here ---

Note the use of "" as the 3rd arg to gensub().  I'm sure that was in the
original (i.e., whoever originally wrote and posted the script had that).
The funny thing is that this script worked just fine right up until I
compiled and started using 4.1.3, which as we've seen now generates a
warning for that usage.  So, I had to change it to use 1 instead of "".

-- 
The scent of awk programmers is a lot more attractive to women than
the scent of perl programmers.

      (Mike Brennan, quoted in the "GAWK" manual)

--- Cut Here ---

0
gazelle
9/5/2015 5:23:34 PM
Reply: