f



Using system() to execute complex shell commands within awk's code, want to simplify the code with in system().

Hi all,

See my following codes:

------------------------------------------------
# extract two fields from the csv list file -- a name and
# a base64 encoded config file, save each of the contained config
# files as plain text in a file
curl --max-time $curl_max_time_csv http://130.158.6.81/api/iphone/ |
awk -F $'\r|,' -v ts=$( date +%s ) '/^vpn/ { printf "%s", $(NF-1) | 
"base64 -d > ."$1"_"$7"_"ts }'
# mimic the ``sed -i'' to replace the job done by dos2unix:
awk -i inplace '{ sub(/\r$/,""); print }' .vpn*

# renanme each config file using specific values
# from the respective config files
awk '
  /^remote / { ip = $2; port = $3 } 
  /^proto / { proto = $2 }
  ENDFILE {
  # only keep the config files for servers which support tcp protocol,
  # and the corresponding config file hasn't been retrieved till now: 
  delete a; split(FILENAME,a,"_")
  if (proto == "tcp" && system( "[ $( find . -type f -regextype posix-
extended -regex .*/vpngate_"ip"_"proto"_"port".ovpn_"a[length(a)-1]"_[1-9]
[0-9]*$ | wc -l ) -eq 0 ]" ) == 0) 
    system( "mv "FILENAME" vpngate_"ip"_"proto"_"port".ovpn_"a[length
(a)-1]"_"a[length(a)] ) 
  else
    system( "rm "FILENAME ) 
}' .vpn* 
------------------------------------------------

In my above codes, I use a system() function to execute complex shell 
commands within awk's code. It seems ugly though it works. I want to 
simplify the code with in system(). 

Any notes/hints?

Regards
-- 
..: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.
0
Hongyi
11/23/2016 4:30:32 AM
comp.lang.awk 3450 articles. 0 followers. Post Follow

16 Replies
379 Views

Similar Articles

[PageSpeed] 27

On Wed, 23 Nov 2016 04:30:32 +0000, Hongyi Zhao wrote:
> awk '
> [...]
>   # and the corresponding config file hasn't been retrieved till now:

The single quote will trigger error, I've changed the code into follows:

awk $'
[...]
# and the corresponding config file hasn\'t been retrieved till now:

Regards
-- 
..: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.
0
Hongyi
11/23/2016 4:49:58 AM
Le mercredi 23 novembre 2016 05:30:33 UTC+1, Hongyi Zhao a =C3=A9crit=C2=A0=
:
> Hi all,
>=20
> See my following codes:
>=20
> ------------------------------------------------
> # extract two fields from the csv list file -- a name and
> # a base64 encoded config file, save each of the contained config
> # files as plain text in a file
> curl --max-time $curl_max_time_csv http://130.158.6.81/api/iphone/ |
> awk -F $'\r|,' -v ts=3D$( date +%s ) '/^vpn/ { printf "%s", $(NF-1) |=20
> "base64 -d > ."$1"_"$7"_"ts }'
> # mimic the ``sed -i'' to replace the job done by dos2unix:
> awk -i inplace '{ sub(/\r$/,""); print }' .vpn*
>=20
> # renanme each config file using specific values
> # from the respective config files
> awk '
>   /^remote / { ip =3D $2; port =3D $3 }=20
>   /^proto / { proto =3D $2 }
>   ENDFILE {
>   # only keep the config files for servers which support tcp protocol,
>   # and the corresponding config file hasn't been retrieved till now:=20
>   delete a; split(FILENAME,a,"_")
>   if (proto =3D=3D "tcp" && system( "[ $( find . -type f -regextype posix=
-
> extended -regex .*/vpngate_"ip"_"proto"_"port".ovpn_"a[length(a)-1]"_[1-9=
]
> [0-9]*$ | wc -l ) -eq 0 ]" ) =3D=3D 0)=20
>     system( "mv "FILENAME" vpngate_"ip"_"proto"_"port".ovpn_"a[length
> (a)-1]"_"a[length(a)] )=20
>   else
>     system( "rm "FILENAME )=20
> }' .vpn*=20
> ------------------------------------------------
>=20
> In my above codes, I use a system() function to execute complex shell=20
> commands within awk's code. It seems ugly though it works. I want to=20
> simplify the code with in system().=20
>=20
> Any notes/hints?
>=20
> Regards
> --=20
> .: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.

You could write with AWK only the commands to be executed and
then excecute them within the shell, not within AWK by system.
0
Marc
11/23/2016 9:29:45 AM
On Wed, 23 Nov 2016 01:29:45 -0800, Marc de Bourget wrote:

> You could write with AWK only the commands to be executed and then
> excecute them within the shell, not within AWK by system.

Do you mean using AWK to construct the commands/arguments with which 
feeds to shell?

Regards
-- 
..: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.
0
Hongyi
11/23/2016 12:09:27 PM
Le mercredi 23 novembre 2016 13:09:28 UTC+1, Hongyi Zhao a =C3=A9crit=C2=A0=
:
> On Wed, 23 Nov 2016 01:29:45 -0800, Marc de Bourget wrote:
>=20
> > You could write with AWK only the commands to be executed and then
> > excecute them within the shell, not within AWK by system.
>=20
> Do you mean using AWK to construct the commands/arguments with which=20
> feeds to shell?
>=20
> Regards
> --=20
> .: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.

Yes. However, I have no clue about Linux. Probably you have something
like .bat files to execute the commands similar to Windows on Linux.
0
Marc
11/23/2016 12:59:15 PM
On 23.11.2016 13:59, Marc de Bourget wrote:
> Le mercredi 23 novembre 2016 13:09:28 UTC+1, Hongyi Zhao a écrit :
>> On Wed, 23 Nov 2016 01:29:45 -0800, Marc de Bourget wrote:
>>
>>> You could write with AWK only the commands to be executed and then
>>> excecute them within the shell, not within AWK by system.
>>
>> Do you mean using AWK to construct the commands/arguments with which 
>> feeds to shell?
>>
>> Regards
>> -- 
>> .: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.
> 
> Yes. However, I have no clue about Linux. Probably you have something
> like .bat files to execute the commands similar to Windows on Linux.

Of course you can write a shell (~ bat|cmd) command file on Unix systems,
but usually you'd just pipe the created commands into a shell

  awk '...{ print "whatever shell commands" }...'  |  sh


Janis

0
Janis
11/23/2016 3:06:42 PM
On Wed, 23 Nov 2016 16:06:42 +0100, Janis Papanagnou wrote:

> Of course you can write a shell (~ bat|cmd) command file on Unix
> systems,
> but usually you'd just pipe the created commands into a shell
> 
>   awk '...{ print "whatever shell commands" }...'  |  sh

For more complex case, the ``sh'' can be substituted by complex shell 
commands flow constructed based on awk's output. Say, for my case, maybe 
the following is more appropriate:

awk '...{ print "whatever shell commands" }...'  |
while IFS= read -r line; do
  ...
done

Regards

> 
> 
> Janis





-- 
..: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.
0
Hongyi
11/24/2016 2:37:30 AM
On 24.11.2016 03:37, Hongyi Zhao wrote:
> On Wed, 23 Nov 2016 16:06:42 +0100, Janis Papanagnou wrote:
> 
>> Of course you can write a shell (~ bat|cmd) command file on Unix
>> systems,
>> but usually you'd just pipe the created commands into a shell
>>
>>   awk '...{ print "whatever shell commands" }...'  |  sh
> 
> For more complex case, the ``sh'' can be substituted by complex shell 
> commands flow constructed based on awk's output. Say, for my case, maybe 
> the following is more appropriate:
> 
> awk '...{ print "whatever shell commands" }...'  |
> while IFS= read -r line; do
>   ...
> done

Yes, but note that shell loops are inherently slow (compared to tools
like awk, perl, etc.). So if every awk 'print' output is related to
shell's 'line' variable you can probably do the loop processing already
in the awk instance. And there's generally the option (where it fits)
to cascade awk calls: awk '...' | awk '...' and avoid shell loops this
way. (With small data set sizes shell loops are okay, of course.)

Janis

> 
> Regards
> 
>>
>>
>> Janis
> 
> 
> 
> 
> 

0
Janis
11/24/2016 7:35:40 AM
On Thu, 24 Nov 2016 08:35:40 +0100, Janis Papanagnou wrote:

> Yes, but note that shell loops are inherently slow (compared to tools
> like awk, perl, etc.). So if every awk 'print' output is related to
> shell's 'line' variable you can probably do the loop processing already
> in the awk instance. And there's generally the option (where it fits) to
> cascade awk calls: awk '...' | awk '...' and avoid shell loops this way.
> (With small data set sizes shell loops are okay, of course.)
> 
> Janis

Thanks a lot for your notes.

Regards
-- 
..: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.
0
Hongyi
11/24/2016 2:23:19 PM
On Thu, 24 Nov 2016 08:35:40 +0100, Janis Papanagnou wrote:

> Yes, but note that shell loops are inherently slow (compared to tools
> like awk, perl, etc.). So if every awk 'print' output is related to
> shell's 'line' variable you can probably do the loop processing already
> in the awk instance. And there's generally the option (where it fits) to
> cascade awk calls: awk '...' | awk '...' and avoid shell loops this way.
> (With small data set sizes shell loops are okay, of course.)
> 
> Janis

As for awk language/utility, the most shortcomings are lack of plenty of 
build-in modules/functions like base64, sha, etc., while for the case of 
python/perl, they have so many build-in modules/functions for using in-
place without borrowing the corresponding tools from shell.

As a result, when using awk, we often invoking the tools/utilities from 
the OS, say, by using the ``system("cmd") and ``print "blabla" | cmd'', 
which make the awk more inefficiently and more hard to use than python/
perl for large/huge data analysis job.

Regards
-- 
..: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.
0
Hongyi
11/24/2016 3:09:40 PM
On 11/24/2016 9:09 AM, Hongyi Zhao wrote:
> On Thu, 24 Nov 2016 08:35:40 +0100, Janis Papanagnou wrote:
>
>> Yes, but note that shell loops are inherently slow (compared to tools
>> like awk, perl, etc.). So if every awk 'print' output is related to
>> shell's 'line' variable you can probably do the loop processing already
>> in the awk instance. And there's generally the option (where it fits) to
>> cascade awk calls: awk '...' | awk '...' and avoid shell loops this way.
>> (With small data set sizes shell loops are okay, of course.)
>>
>> Janis
>
> As for awk language/utility, the most shortcomings are lack of plenty of
> build-in modules/functions like base64, sha, etc., while for the case of
> python/perl, they have so many build-in modules/functions for using in-
> place without borrowing the corresponding tools from shell.
>
> As a result, when using awk, we often invoking the tools/utilities from
> the OS, say, by using the ``system("cmd") and ``print "blabla" | cmd'',
> which make the awk more inefficiently and more hard to use than python/
> perl for large/huge data analysis job.
>
> Regards
>

No, you just don't know how/when to use awk. Read the book Effective Awk 
Programming, 4th Edition by Arnold Robbins before making sweeping statements 
about something you simply do not understand.

     Ed.
0
Ed
11/24/2016 4:06:06 PM
In article <o17338$2kj$1@dont-email.me>,
Ed Morton  <mortonspam@gmail.com> wrote:
>On 11/24/2016 9:09 AM, Hongyi Zhao trolled:
....
>> of most etc., build-in tools for modules/functions they without
>> shortcomings while modules/functions from awk like have borrowing are
>> for for shell.  language/utility, base64, so the of lack case the in-
>> using plenty of the sha, many corresponding As build-in python/perl, place

>> from cmd'', than awk, the inefficiently job.  a OS, make for python/
>> we ``system("cmd") and result, say, the large/huge often and more when
>> by awk data the invoking "blabla" ``print to hard tools/utilities |
>> use using using more analysis As the which perl

>No, you just don't know how/when to use awk. Read the book Effective Awk
>Programming, 4th Edition by Arnold Robbins before making sweeping
>statements about something you simply do not understand.

YHBT

-- 
(Cruz certainly has an odd face) ... it looks like someone sewed pieces of a
waterlogged Reagan mask together at gunpoint ...

http://www.rollingstone.com/politics/news/how-america-made-donald-trump-unstoppable-20160224
0
gazelle
11/24/2016 4:35:28 PM
On 24.11.2016 16:09, Hongyi Zhao wrote:
> 
> As for awk language/utility, the most shortcomings are lack of plenty of 
> build-in modules/functions like base64, sha, etc., while for the case of 
> python/perl, they have so many build-in modules/functions for using in-
> place without borrowing the corresponding tools from shell.
> 
> As a result, when using awk, we often invoking the tools/utilities from 
> the OS, say, by using the ``system("cmd") and ``print "blabla" | cmd'', 
> which make the awk more inefficiently and more hard to use than python/
> perl for large/huge data analysis job.

You may have missed awk's application domain. Awk is not a Swiss Army Knife.
You are probably looking for another tool. You should consider changing your
tool for the tasks you do.

Janis

> 
> Regards
> 

0
Janis
11/24/2016 6:00:57 PM
"Janis Papanagnou" <janis_papanagnou@hotmail.com> ha scritto nel messaggio 
news:o179sq$6lv$1@news-1.m-online.net...
>
> You may have missed awk's application domain. Awk is not a Swiss Army 
> Knife.
> You are probably looking for another tool. You should consider changing 
> your
> tool for the tasks you do.

I do not agree with you. With proper usage of co-processing you may do 
almost anything you need without learning other apparently easier and more 
convenient languages. 


0
MadSharker
12/11/2016 9:44:18 AM
On 11.12.2016 10:44, MadSharker wrote:
> "Janis Papanagnou" <janis_papanagnou@hotmail.com> ha scritto nel messaggio 
> news:o179sq$6lv$1@news-1.m-online.net...
>>
>> You may have missed awk's application domain. Awk is not a Swiss Army 
>> Knife.
>> You are probably looking for another tool. You should consider changing 
>> your
>> tool for the tasks you do.
> 
> I do not agree with you. With proper usage of co-processing you may do 
> almost anything you need without learning other apparently easier and more 
> convenient languages. 

This assumes GNU awk! And if you have to rely on many external processes you
don't rely on the language features; that typically means inefficient and/or
clumsy programs. YMMV, of course. Usually if I have many shell/pipe connected
external programs it's better to switch perspective and do that in shell with
help of awk only for the text processing subtasks, i.e. the tasks awk has been
designed for.

Janis

0
Janis
12/11/2016 9:55:10 AM
On 12/11/2016 3:44 AM, MadSharker wrote:
> "Janis Papanagnou" <janis_papanagnou@hotmail.com> ha scritto nel messaggio
> news:o179sq$6lv$1@news-1.m-online.net...
>>
>> You may have missed awk's application domain. Awk is not a Swiss Army
>> Knife.
>> You are probably looking for another tool. You should consider changing
>> your
>> tool for the tasks you do.
>
> I do not agree with you. With proper usage of co-processing you may do
> almost anything you need without learning other apparently easier and more
> convenient languages.

The same is true of assembly code and you can drive screws into wood with a 
hammer. Just because you CAN do something doesn't make it the best, or even a 
reasonable, approach. Just use the right tool for each job.

	Ed.
0
Ed
12/11/2016 2:27:55 PM
"Janis Papanagnou" <janis_papanagnou@hotmail.com> ha scritto nel messaggio 
news:o2j7pu$28h$1@news-1.m-online.net...
>
> This assumes GNU awk! And if you have to rely on many external processes 
> you
> don't rely on the language features; that typically means inefficient 
> and/or
> clumsy programs.

Ok, ok, maybe I'm a Gawk lunatic fan. :-)
But, as far as I am concerned, I do not see many differences between calling 
an external program to solve jobs like charset conversion or web page 
retrieval and calling a Cpan function with all its peculiarities. 


0
MadSharker
12/12/2016 5:16:42 PM
Reply: