f



How to use only one time of awk invoking to do this thing: $ echo abcde | awk '{ system("rev<<<"$0) }' | awk '{a=$0}'?

Hi all,

How to use only one time of awk invoking to do this thing:

$ echo abcde | awk '{ system("rev<<<"$0) }' | awk '{a=$0}'?

Regards
-- 
..: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.
0
Hongyi
11/19/2016 12:33:07 PM
comp.lang.awk 3450 articles. 0 followers. Post Follow

15 Replies
235 Views

Similar Articles

[PageSpeed] 17

On 19.11.2016 13:33, Hongyi Zhao wrote:
> Hi all,
> 
> How to use only one time of awk invoking to do this thing:
> 
> $ echo abcde | awk '{ system("rev<<<"$0) }' | awk '{a=$0}'?

  echo abcde | rev | awk '{a=$0}'

But you should probably better tell us what you are actually trying to do.

Janis

> 
> Regards
> 

0
Janis
11/19/2016 2:33:56 PM
On Sat, 19 Nov 2016 15:33:56 +0100, Janis Papanagnou wrote:

> But you should probably better tell us what you are actually trying to
> do.

Currently I use the following codes for my job:

# Obtain the content of OpenVPN_ConfigData_Base64 
# and store them separately, one file for each OpenVPN server's config: 

curl -s http://www.vpngate.net/api/iphone/ |
awk -F $'\r|,' '/^vpn/ { printf "%s", $(NF-1) | "base64 -d | dos2unix 
>"$1 }'

# Based on the OpenVPN config file's content,
# rename its name into this form: vpngate_ip_proto_port.ovpn
for i in vpn*; do
  mv $i $( awk '/^remote / { ip = $2; port = $3 }
                /^proto / { proto = $2 }
               END { print "vpngate_"ip"_"proto"_"port".ovpn" }' $i )
done 

I want to do the above job more concisely, say, reconstruct the code into 
the following form:

curl -s http://www.vpngate.net/api/iphone/ |
awk '
 blabla
'

Regards

> 
> Janis





-- 
..: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.
0
Hongyi
11/19/2016 3:31:46 PM
On 19.11.2016 16:31, Hongyi Zhao wrote:
> On Sat, 19 Nov 2016 15:33:56 +0100, Janis Papanagnou wrote:
> 
>> But you should probably better tell us what you are actually trying to
>> do.
> 
> Currently I use the following codes for my job:

It seems your progranm wants to
 *  extract two fields from a file, a name and an encoded config file
 *  save each of the contained config files as plain text in a file
 *  renanme each config file using specific values from the respective
      config files

Is that correct? - You may then notice that you magically need to know the
data from the config files before you can use the redirection on the file
to create it. That means; either store it temporarily in memory and write
it when you're finished with each config file, or do a two-pass processing,
or do it in two steps (as you've done), or create a temporary file that you
rename after the respective config file is parsed. Pick the one you like.

> 
> # Obtain the content of OpenVPN_ConfigData_Base64 
> # and store them separately, one file for each OpenVPN server's config: 
> 
> curl -s http://www.vpngate.net/api/iphone/ |
> awk -F $'\r|,' '/^vpn/ { printf "%s", $(NF-1) | "base64 -d | dos2unix 
>> "$1 }'
> 
> # Based on the OpenVPN config file's content,
> # rename its name into this form: vpngate_ip_proto_port.ovpn
> for i in vpn*; do
>   mv $i $( awk '/^remote / { ip = $2; port = $3 }
>                 /^proto / { proto = $2 }
>                END { print "vpngate_"ip"_"proto"_"port".ovpn" }' $i )
> done 

To simplify your code (in case you stay with your two-step processing, and
assuming you use GNU awk) you can also avoid the shell loop and let awk
read the file list.

  awk '
    /^remote / { ip = $2; port = $3 } ; /^proto / { proto = $2 }
    ENDFILE { print "mv "FILENAME" vpngate_"ip"_"proto"_"port".ovpn"}
  ' vpn* | sh

Thereby you also avoid N instances of awk and N subshells in your shell
loop. (Externalizing the 'sh' call is done here to avoid N system("mv ...")
subshells in awk.)

Janis

> 
> I want to do the above job more concisely, say, reconstruct the code into 
> the following form:
> 
> curl -s http://www.vpngate.net/api/iphone/ |
> awk '
>  blabla
> '
> 
> Regards
> 
>>
>> Janis
> 
> 
> 
> 
> 

0
Janis
11/19/2016 4:51:46 PM
On 19.11.2016 16:31, Hongyi Zhao wrote:
> [...]
> curl -s http://www.vpngate.net/api/iphone/ |
> awk -F $'\r|,' '/^vpn/ { printf "%s", $(NF-1) | "base64 -d | dos2unix > "$1 }'
> [...]

BTW, since I noticed that I seem to not have dos2unix on my box; if all you
want to do is removing a spurious CR at the end of the line you can do that
in your awk program with  sub(/\r$/,"")  and avoid another external program.

Janis

0
Janis
11/19/2016 4:56:20 PM
On Sat, 19 Nov 2016 17:56:20 +0100, Janis Papanagnou wrote:

> BTW, since I noticed that I seem to not have dos2unix on my box; if all
> you want to do is removing a spurious CR at the end of the line you can
> do that in your awk program with  sub(/\r$/,"")  and avoid another
> external program.

Thanks a lot for your hints.

Regards
> 
> Janis





-- 
..: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.
0
Hongyi
11/20/2016 1:30:38 AM
On Sat, 19 Nov 2016 17:56:20 +0100, Janis Papanagnou wrote:

> On 19.11.2016 16:31, Hongyi Zhao wrote:
>> [...]
>> curl -s http://www.vpngate.net/api/iphone/ |
>> awk -F $'\r|,' '/^vpn/ { printf "%s", $(NF-1) | "base64 -d | dos2unix >
>> "$1 }'
>> [...]
> 
> BTW, since I noticed that I seem to not have dos2unix on my box; if all
> you want to do is removing a spurious CR at the end of the line you can
> do that in your awk program with  sub(/\r$/,"")  and avoid another
> external program.

I do a further try based on your above hints, but the difficult for this 
case lies in that:

[1] The end-of-line character of the original file, i.e., ``http://
www.vpngate.net/api/iphone/'' is ms-windows type.

[2] The based64 encoded data is still ms-windows type in its decoded / 
original form.

I want to remove the CR at the end of the line from the output of the 
``base64 -d''.  So, I don't know where to use sub(/\r$/,"") in my awk 
code.

Regards


> 
> Janis





-- 
..: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.
0
Hongyi
11/20/2016 1:43:30 AM
On Sat, 19 Nov 2016 17:56:20 +0100, Janis Papanagnou wrote:

> On 19.11.2016 16:31, Hongyi Zhao wrote:
>> [...]
>> curl -s http://www.vpngate.net/api/iphone/ |
>> awk -F $'\r|,' '/^vpn/ { printf "%s", $(NF-1) | "base64 -d | dos2unix >
>> "$1 }'
>> [...]
> 
> BTW, since I noticed that I seem to not have dos2unix on my box; if all
> you want to do is removing a spurious CR at the end of the line you can
> do that in your awk program with  sub(/\r$/,"")  and avoid another
> external program.

Based on above hints, I can only find the following code without using 
dos2unix, 

curl -s http://www.vpngate.net/api/iphone/ |
awk -F $'\r|,' '/^vpn/ { printf "%s", $(NF-1) | "base64 -d"}' |
awk 'sub(/\r$/,"")'

But this will concatenate all of the openvpn config files into one file.

Regards

> 
> Janis





-- 
..: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.
0
Hongyi
11/20/2016 3:55:16 AM
On Sat, 19 Nov 2016 17:51:46 +0100, Janis Papanagnou wrote:

> It seems your progranm wants to
>  *  extract two fields from a file, a name and an encoded config file * 
>  save each of the contained config files as plain text in a file * 
>  renanme each config file using specific values from the respective
>       config files
> 
> Is that correct? 

Yes, this is just what I want to do.

> - You may then notice that you magically need to know
> the data from the config files before you can use the redirection on the
> file to create it. That means; either store it temporarily in memory and
> write it when you're finished with each config file, or do a two-pass
> processing,
> or do it in two steps (as you've done), or create a temporary file that
> you rename after the respective config file is parsed. Pick the one you
> like.

Among all of the methods for my job, I really like the method based on 
coprocess of gawk.  I have read the doc here:

https://www.gnu.org/software/gawk/manual/html_node/Two_002dway-
I_002fO.html

But still cann't figure out the implementation based on this method for 
my issue.

Regards
-- 
..: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.
0
Hongyi
11/20/2016 6:39:15 AM
On 20.11.2016 07:39, Hongyi Zhao wrote:
> On Sat, 19 Nov 2016 17:51:46 +0100, Janis Papanagnou wrote:
> 
>> It seems your progranm wants to
>>  *  extract two fields from a file, a name and an encoded config file * 
>>  save each of the contained config files as plain text in a file * 
>>  renanme each config file using specific values from the respective
>>       config files
>>
>> Is that correct? 
> 
> Yes, this is just what I want to do.
> 
>> - You may then notice that you magically need to know
>> the data from the config files before you can use the redirection on the
>> file to create it. That means; either store it temporarily in memory and
>> write it when you're finished with each config file, or do a two-pass
>> processing,
>> or do it in two steps (as you've done), or create a temporary file that
>> you rename after the respective config file is parsed. Pick the one you
>> like.
> 
> Among all of the methods for my job, I really like the method based on 
> coprocess of gawk.  I have read the doc here:
> 
> https://www.gnu.org/software/gawk/manual/html_node/Two_002dway-
> I_002fO.html
> 
> But still cann't figure out the implementation based on this method for 
> my issue.

Well, I considered showing you a coprocess based way as well but abstained
because of two reasons; one is the already mentioned general fact that you
want to create a file but need data from the same file in advance, and a
coprocess won't help you here without any additional means as I mentioned.
And  the second is even a bit nastier (and off-topic here); there's issues
with the coprocess mechanism, e.g. in case the coprocess operates fully
buffered (as opposed to line buffered), so we'd need yet more off-topic
details (like, e.g., pty or screen) to make some tools work.

You can experiment with coprocesses and various external filter tools; try
it with cat, bc, rev, etc., to observe differences. How to use coprocesses
is described in the GNU awk manual. Or you can use this minimalistic script
for tests (for production code getline needs some error handling)

  awk -v cmd=${1:-cat} '
    { print |& cmd ; cmd |& getline res
      print "#", $0, "->", res }
  '


Janis

> 
> Regards
> 

0
Janis
11/20/2016 9:05:28 AM
On 20.11.2016 02:43, Hongyi Zhao wrote:
> On Sat, 19 Nov 2016 17:56:20 +0100, Janis Papanagnou wrote:
> 
>> On 19.11.2016 16:31, Hongyi Zhao wrote:
>>> [...]
>>> curl -s http://www.vpngate.net/api/iphone/ |
>>> awk -F $'\r|,' '/^vpn/ { printf "%s", $(NF-1) | "base64 -d | dos2unix >
>>> "$1 }'
>>> [...]
>>
>> BTW, since I noticed that I seem to not have dos2unix on my box; if all
>> you want to do is removing a spurious CR at the end of the line you can
>> do that in your awk program with  sub(/\r$/,"")  and avoid another
>> external program.
> 
> I do a further try based on your above hints, but the difficult for this 
> case lies in that:
> 
> [1] The end-of-line character of the original file, i.e., ``http://
> www.vpngate.net/api/iphone/'' is ms-windows type.
> 
> [2] The based64 encoded data is still ms-windows type in its decoded / 
> original form.
> 
> I want to remove the CR at the end of the line from the output of the 
> ``base64 -d''.  So, I don't know where to use sub(/\r$/,"") in my awk 
> code.

It depends on the actual calling structure you have chosen - I've posted
a couple. This may be, e.g., in the second awk process, the one that gets
the base64-decoded data...

  awk '{... "base64 -d" ...}' | ... | awk '{ sub(/\r$/,"") ; ... }'

Or generally; you do it in the awk process that generate the file. (If
you let the shell do that file creation awk of course won't help you here,
but you have the choice, as said upthread.)

Janis

> 
> Regards
> 
> 
>>
>> Janis
> 
> 
> 
> 
> 

0
Janis
11/20/2016 9:13:24 AM
On 20.11.2016 04:55, Hongyi Zhao wrote:
> On Sat, 19 Nov 2016 17:56:20 +0100, Janis Papanagnou wrote:
> 
>> On 19.11.2016 16:31, Hongyi Zhao wrote:
>>> [...]
>>> curl -s http://www.vpngate.net/api/iphone/ |
>>> awk -F $'\r|,' '/^vpn/ { printf "%s", $(NF-1) | "base64 -d | dos2unix >
>>> "$1 }'
>>> [...]
>>
>> BTW, since I noticed that I seem to not have dos2unix on my box; if all
>> you want to do is removing a spurious CR at the end of the line you can
>> do that in your awk program with  sub(/\r$/,"")  and avoid another
>> external program.
> 
> Based on above hints, I can only find the following code without using 
> dos2unix, 
> 
> curl -s http://www.vpngate.net/api/iphone/ |
> awk -F $'\r|,' '/^vpn/ { printf "%s", $(NF-1) | "base64 -d"}' |
> awk 'sub(/\r$/,"")'
> 
> But this will concatenate all of the openvpn config files into one file.

You still have to implement one of the options you have to work on the
individual files. I said:
>>>> [...] You may then notice that you magically need to know the
>>>> data from the config files before you can use the redirection on the file
>>>> to create it. That means; either store it temporarily in memory and write
>>>> it when you're finished with each config file, or do a two-pass processing,
>>>> or do it in two steps (as you've done), or create a temporary file that you
>>>> rename after the respective config file is parsed. Pick the one you like.

> 
> Regards
> 
>>
>> Janis
> 
> 
> 
> 
> 

0
Janis
11/20/2016 9:18:02 AM
On Sat, 19 Nov 2016 17:51:46 +0100, Janis Papanagnou wrote:

>   awk '
>     /^remote / { ip = $2; port = $3 } ; /^proto / { proto = $2 }

You use semicolon here just for improving the readability?  I mean, 
remove the `;' here will make no differences.

Regards

>  ENDFILE { print "mv "FILENAME" vpngate_"ip"_"proto"_"port".ovpn"}
>   ' vpn* | sh





-- 
..: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.
0
Hongyi
11/20/2016 12:12:02 PM
On Sun, 20 Nov 2016 10:05:28 +0100, Janis Papanagnou wrote:
> [...]
> You can experiment with coprocesses and various external filter tools;
> try it with cat, bc, rev, etc., to observe differences. How to use
> coprocesses is described in the GNU awk manual. Or you can use this
> minimalistic script for tests (for production code getline needs some
> error handling)
> 
>   awk -v cmd=${1:-cat} '
>     { print |& cmd ; cmd |& getline res
>       print "#", $0, "->", res }
>   '

Based on all of the previous discussions on this topic, I use the 
following code for my job now:

#  extract two fields from a file, a name and an encoded config file
#  save each of the contained config files as plain text in a file
curl -s http://www.vpngate.net/api/iphone/ |
awk -F $'\r|,' '/^vpn/ { printf "%s", $(NF-1) | "base64 -d >" $1 }'
 
#  renanme each config file using specific values from the respective
      config files
awk '
    { sub(/\r$/,"") } ; /^remote / { ip = $2; port = $3 } ; /^proto / 
{ proto = $2 }
    ENDFILE { print "mv "FILENAME" vpngate_"ip"_"proto"_"port".ovpn"}
  ' vpn* | sh

Though I don't think it's graceful, it don't use the dos2unix and the 
bash/shell for-loops now.

Thanks a lot.

Regards

> 
> 
> Janis





-- 
..: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.
0
Hongyi
11/20/2016 12:31:15 PM
On Sun, 20 Nov 2016 12:31:15 +0000, Hongyi Zhao wrote:

> #  renanme each config file using specific values from the respective
>       config files
> awk '
>     { sub(/\r$/,"") } ; /^remote / { ip = $2; port = $3 } ; /^proto /
> { proto = $2 }
>     ENDFILE { print "mv "FILENAME" vpngate_"ip"_"proto"_"port".ovpn"}
>   ' vpn* | sh

Some corrections on my above code: when don't use the dos2unix, the 
similar operation with awk can be done as follows:

# mimic the ``sed -i'':
awk -i inplace '{ sub(/\r$/,""); print }' vpn*
# renanme each config file using specific values
# from the respective config files
awk '
 /^remote / { ip = $2; port = $3 } ; /^proto / { proto = $2 }
 ENDFILE { print "mv "FILENAME" vpngate_"ip"_"proto"_"port".ovpn"}
' vpn* | sh

Regards
-- 
..: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.
0
Hongyi
11/20/2016 2:53:43 PM
On 20.11.2016 13:12, Hongyi Zhao wrote:
> On Sat, 19 Nov 2016 17:51:46 +0100, Janis Papanagnou wrote:
> 
>>   awk '
>>     /^remote / { ip = $2; port = $3 } ; /^proto / { proto = $2 }
> 
> You use semicolon here just for improving the readability?  I mean, 
> remove the `;' here will make no differences.

According to the standard syntax (as someone pointed out a couple of
years ago) it's necessary. But it had also been pointed out that all
existing awk variants would accept the syntax without a semicolon.

Usually I don't use it. (In this case I added it to more easily see
where you'd split the two commands.) For readability better put each
command on a line of its own. In this case I joined the lines because
the focus was on the overall structure how to compose the programs,
not what specific patterns/actions are actually used.

Janis

> 
> Regards
> 
>>  ENDFILE { print "mv "FILENAME" vpngate_"ip"_"proto"_"port".ovpn"}
>>   ' vpn* | sh
> 
> 
> 
> 
> 

0
Janis
11/22/2016 8:15:20 AM
Reply: