I need to query a LDAP server, then process the putput with out using
temp files.

I have have a working script which uses system() to call shell scripts
which then send the output to a file.
But as the script gets more complex, it takes too long to run.
Basically I need to go through the LDAP database and identify all the
groups and its members.  Next I need to create a loop to make sure I
find all the nested groups, and thats were this method becomes to slow
to be effective.

Thanks
Sal

BEGIN {
    excludefile = "exclude-groups.txt"
    i=0
    while( (getline line < excludefile) > 0 ) {
        exclude[i] = line
        i++
    }
    close( excludefile )

    RS = ""
    FS = "\n"
    x=0
}

{
    skip_record = "false"
    x++

    # test if group is found in exclude list
    for(j=0; j<=i; j++) {
        if( $1 == exclude[j] ) {
            skip_record = "true"
            x--
            break
        }
    }

    if( skip_record == "false" )
        process[x] = $1
}

END {
    # delete existing "get-domino-members.cmd" file
    system("del get-domino-members.cmd")

    # create script file to query Domino for member lists
    for( j=0; j <= x; j++ ) {
        printf "ldapsearch -p 389 -D \"userid\" -w password -h server
\"%s\" -s base \"objectClass=*\" member >> domino-members.txt \n",
process[j] > "get-domino-members.cmd"
        printf "\n" >> "get-domino-members.cmd"
    }
    printf "\n" >> "get-domino-members.cmd"

    # execute get-domino-members.cmd script to get list of members
    system("get-domino-members.cmd")
}


I have researched this issue for a while, and I'm only confusing
myself further.

On 29    , 22:02, SP <polif...@comcast.net> wrote:
> I need to query a LDAP server, then process the putput with out using
> temp files.
>
> I have have a working script which uses system() to call shell scripts
> which then send the output to a file.
> But as the script gets more complex, it takes too long to run.
> Basically I need to go through the LDAP database and identify all the
> groups and its members.  Next I need to create a loop to make sure I
> find all the nested groups, and thats were this method becomes to slow
> to be effective.
>
> Thanks
> Sal
>
> BEGIN {
>     excludefile = "exclude-groups.txt"
>     i=0
>     while( (getline line < excludefile) > 0 ) {
>         exclude[i] = line
>         i++
>     }
>     close( excludefile )
>
>     RS = ""
>     FS = "\n"
>     x=0
>
> }
>
> {
>     skip_record = "false"
>     x++
>
>     # test if group is found in exclude list
>     for(j=0; j<=i; j++) {
>         if( $1 == exclude[j] ) {
>             skip_record = "true"
>             x--
>             break
>         }
>     }
>
>     if( skip_record == "false" )
>         process[x] = $1
>
> }
>
> END {
>     # delete existing "get-domino-members.cmd" file
>     system("del get-domino-members.cmd")
>
>     # create script file to query Domino for member lists
>     for( j=0; j <= x; j++ ) {
>         printf "ldapsearch -p 389 -D \"userid\" -w password -h server
> \"%s\" -s base \"objectClass=*\" member >> domino-members.txt \n",
> process[j] > "get-domino-members.cmd"
>         printf "\n" >> "get-domino-members.cmd"
>     }
>     printf "\n" >> "get-domino-members.cmd"
>
>     # execute get-domino-members.cmd script to get list of members
>     system("get-domino-members.cmd")
>
> }
>
> I have researched this issue for a while, and I'm only confusing
> myself further.


Where to start?
Oh well, this is what ITYM:

NR == FNR { exclude[$0]++; next } # first file: exclude list

# second file: main input
{
	if ($0 in exclude)
		next

	process[$0]++
}

END {
	# create script file to query Domino for member lists
	for (i in process)
        	printf "ldapsearch -p 389 -D userid -w password -h server \"%s
\" -s base \"objectClass=*\" member >>domino-members.txt\n\n",
process[i] > "get-domino-members.cmd"

	print > "get-domino-members.cmd"
}

Put this in a file name mkquerycmd.awk.
Make another file to call mkquerycmd.awk like:

awk -f mkquerycmd.awk exclude-groups.txt main-input >get-domino-
members.cmd
get-domino-members.cmd

Try this and let us know.
Some input from your files would be of great help.

  Vassilis

On Aug 29, 3:43 pm, Vassilis <F.H.Nova...@gmail.com> wrote:
> On 29    , 22:02, SP <polif...@comcast.net> wrote:
>
>
>
> > I need to query a LDAP server, then process the putput with out using
> > temp files.
>
> > I have have a working script which uses system() to call shell scripts
> > which then send the output to a file.
> > But as the script gets more complex, it takes too long to run.
> > Basically I need to go through the LDAP database and identify all the
> > groups and its members.  Next I need to create a loop to make sure I
> > find all the nested groups, and thats were this method becomes to slow
> > to be effective.
>
> > Thanks
> > Sal
>
> > BEGIN {
> >     excludefile = "exclude-groups.txt"
> >     i=0
> >     while( (getline line < excludefile) > 0 ) {
> >         exclude[i] = line
> >         i++
> >     }
> >     close( excludefile )
>
> >     RS = ""
> >     FS = "\n"
> >     x=0
>
> > }
>
> > {
> >     skip_record = "false"
> >     x++
>
> >     # test if group is found in exclude list
> >     for(j=0; j<=i; j++) {
> >         if( $1 == exclude[j] ) {
> >             skip_record = "true"
> >             x--
> >             break
> >         }
> >     }
>
> >     if( skip_record == "false" )
> >         process[x] = $1
>
> > }
>
> > END {
> >     # delete existing "get-domino-members.cmd" file
> >     system("del get-domino-members.cmd")
>
> >     # create script file to query Domino for member lists
> >     for( j=0; j <= x; j++ ) {
> >         printf "ldapsearch -p 389 -D \"userid\" -w password -h server
> > \"%s\" -s base \"objectClass=*\" member >> domino-members.txt \n",
> > process[j] > "get-domino-members.cmd"
> >         printf "\n" >> "get-domino-members.cmd"
> >     }
> >     printf "\n" >> "get-domino-members.cmd"
>
> >     # execute get-domino-members.cmd script to get list of members
> >     system("get-domino-members.cmd")
>
> > }
>
> > I have researched this issue for a while, and I'm only confusing
> > myself further.
>
> Where to start?
> Oh well, this is what ITYM:
>
> NR == FNR { exclude[$0]++; next } # first file: exclude list
>
> # second file: main input
> {
>         if ($0 in exclude)
>                 next
>
>         process[$0]++
>
> }
>
> END {
>         # create script file to query Domino for member lists
>         for (i in process)
>                 printf "ldapsearch -p 389 -D userid -w password -h server \"%s
> \" -s base \"objectClass=*\" member >>domino-members.txt\n\n",
> process[i] > "get-domino-members.cmd"
>
>         print > "get-domino-members.cmd"
>
> }
>
> Put this in a file name mkquerycmd.awk.
> Make another file to call mkquerycmd.awk like:
>
> awk -f mkquerycmd.awk exclude-groups.txt main-input >get-domino-
> members.cmd
> get-domino-members.cmd
>
> Try this and let us know.
> Some input from your files would be of great help.
>
>   Vassilis

Sorry I was not clear, the existing script is not complete.
The next step is to create LDIF file or files which would recreate the
Domino LDAP structure in Microsoft Active Directory.

I need the to make sure that first all groups which do NOT contain any
groups are created
Then create groups for which the sub-groups already exist.
Finally place the users into the appropriate group.

I thought that by calling the "ldapsearch" command directly from awk
would allow me to process
each member recursively with out the need for all the temp files.

The following is how I call the awk script and the associated data.

main batch file contains:
  call export-domino-groups.cmd
  gawk -f test.awk domino-groups.txt
  call get-domino-members.cmd

script export-domino-groups.cmd
ldapsearch -p 389 -D "UserID" -w Password -h Hostname -b ""
"objectClass=groupOfNames" DN > domino-groups.txt

data file domino-groups.txt [ this file contains about 1,500 entries ]
....
CN=Professional Services - Imaging Specialists - East

CN=Managed Services - System Support

CN=Managed Services - All

CN=Marketing - Clinical Sales Specialist

CN=ME4N - Data Management

CN=Sales - Reporting Administrators
....


script get-domino-members.cmd [created by awk script, there about 500
groups]
....
ldapsearch -p 389 -D "UserID" -w Password -h Hostname "CN=Managed
Services - All" -s base "objectClass=*" member >> domino-members.txt
ldapsearch -p 389 -D "UserID" -w Password -h Hostname "CN=Marketing -
Clinical Sales Specialist" -s base "objectClass=*" member >> domino-
members.txt
....


data file domino-members.txt [the rsulting file has about 35,000
entries]
....
CN=Engineering - Synapse - Support Group
member=CN=Radha Lastname,O=mycompany
member=CN=Rob Lastname,O=mycompany
member=CN=John Lastname,O=mycompany
member=CN=Annie Lastname,O=mycompany
member=CN=Donald Lastname,O=mycompany
member=CN=Edward Lastname,O=mycompany

CN=Engineering - Stamford Group
member=CN=Engineering - Managers
member=CN=Engineering - Synapse - Product Support
member=CN=Engineering - Synapse - Software Development
member=CN=Engineering - Synapse - Support Group
member=CN=Engineering - Synapse Database
member=CN=Engineering - Testing
member=CN=Jane Doe,O=mycompany

....

SP wrote:
> Sorry I was not clear, the existing script is not complete.
> The next step is to create LDIF file or files which would recreate the
> Domino LDAP structure in Microsoft Active Directory.

You assume that "everyone" knows the finer details of this two LDAP 
databases. I don't think this is the case in this group.

Show us a simple exemplary input text file which should go into awk and 
the corresponding output you want to get out of awk. Tell us how and 
where the elements in the input data should end up in the output data. 
Do not assume everyone here knows LDAP, LDAP query syntax, the LDAP data 
interchange format, "groups" (you mean organizational units?), etc.

> I need the to make sure that first all groups which do NOT contain any
> groups are created
> Then create groups for which the sub-groups already exist.
> Finally place the users into the appropriate group.

Consider asking in an LDAP group, e.g. the vendor specific 
microsoft.public.windows.server.active_directory or ibm.software.ldap 
(Domino is from IBM, or?), or general database groups like 
comp.databases, or similar, if there is a way to optimize your LDAP 
queries to get the data in the order you would like to have, instead of 
first requesting all data from the DB, and then sorting it out.

If you need a more tight integration between your query and your 
processing consider "that other text processing tool" P*rl :-). It has 
an LDAP module, so you can directly access LDAP DBs from within your 
script without the need to call external programs.

BR,

/Thomas

On Aug 30, 4:18 am, Thomas Weidenfeller <nob...@ericsson.invalid>
wrote:
> SP wrote:
> > Sorry I was not clear, the existing script is not complete.
> > The next step is to create LDIF file or files which would recreate the
> > Domino LDAP structure in Microsoft Active Directory.
>
> You assume that "everyone" knows the finer details of this two LDAP
> databases. I don't think this is the case in this group.
>
> Show us a simple exemplary input text file which should go into awk and
> the corresponding output you want to get out of awk. Tell us how and
> where the elements in the input data should end up in the output data.
> Do not assume everyone here knows LDAP, LDAP query syntax, the LDAP data
> interchange format, "groups" (you mean organizational units?), etc.
>
> > I need the to make sure that first all groups which do NOT contain any
> > groups are created
> > Then create groups for which the sub-groups already exist.
> > Finally place the users into the appropriate group.
>
> Consider asking in an LDAP group, e.g. the vendor specific
> microsoft.public.windows.server.active_directory or ibm.software.ldap
> (Domino is from IBM, or?), or general database groups like
> comp.databases, or similar, if there is a way to optimize your LDAP
> queries to get the data in the order you would like to have, instead of
> first requesting all data from the DB, and then sorting it out.
>
> If you need a more tight integration between your query and your
> processing consider "that other text processing tool" P*rl :-). It has
> an LDAP module, so you can directly access LDAP DBs from within your
> script without the need to call external programs.
>
> BR,
>
> /Thomas

Thanks for suggestions, I would like to give this one more try before
looking at other languages.

The external command "ldapsearch" is given the name of a group and it
returns a list of members:

the command:
ldapsearch ... "CN=Engineering - Stamford Group" ... member

returns:
CN=Engineering - Stamford Group
member=CN=Engineering - Managers
member=CN=Engineering - Synapse - Product Support
member=CN=Engineering - Synapse - Software Development
member=CN=Engineering - Synapse - Support Group
member=CN=Engineering - Synapse Database
member=CN=Engineering - Testing
member=CN=Jane Doe,O=mycompany

I'm trying to call the command and then process each line of output, I
found a refernece to
the |& pipe command, and need an example on how its used.

Thank you for your feddback.

"SP" <polifemo@comcast.net> wrote in message
news:1188414132.994573.313820@g4g2000hsf.googlegroups.com...
> I have have a working script

Okay, I'll take your word for it.

> But as the script gets more complex, it takes too long to run.

Again, I'll take your word for it.

> BEGIN {
>     excludefile = "exclude-groups.txt"
>     i=0
>     while( (getline line < excludefile) > 0 ) {
>         exclude[i] = line
>         i++
>     }
>     close( excludefile )
>
>     RS = ""
>     FS = "\n"
>     x=0
> }
>
> {
>     skip_record = "false"
>     x++
>
>     # test if group is found in exclude list
>     for(j=0; j<=i; j++) {
>         if( $1 == exclude[j] ) {
>             skip_record = "true"
>             x--
>             break
>         }
>     }
>
>     if( skip_record == "false" )
>         process[x] = $1
> }
>
> END {
>     # delete existing "get-domino-members.cmd" file
>     system("del get-domino-members.cmd")
>
>     # create script file to query Domino for member lists
>     for( j=0; j <= x; j++ ) {
>         printf "ldapsearch -p 389 -D \"userid\" -w password -h server
> \"%s\" -s base \"objectClass=*\" member >> domino-members.txt \n",
> process[j] > "get-domino-members.cmd"
>         printf "\n" >> "get-domino-members.cmd"
>     }
>     printf "\n" >> "get-domino-members.cmd"
>
>     # execute get-domino-members.cmd script to get list of members
>     system("get-domino-members.cmd")
> }

Vassilis pretty much solved your problem of speed-up, though you may not
recognize that. Except I'd write the second file processing as:

{
    if ( !($1 in exclude) )
        process[$1]++
}

which should be logically equivalent but fits my sense of 'how it ought to
be' a little better.

The key to speeding your process up in AWK is to drop the idea of using
integer-indexed arrays in which you store data at each index. Instead use
the data itself as indices and store any throw-away value whatsoever at each
index (in this case, the value '1' is stored at each index, but never used
anywhere).

This allows you to use the 'in' keyword to quickly check if data in the
second file occurs anywhere in your exclude list and a 'for ( .. in .. )'
loop to create your commands in the END part.

Oh, and in the END part I might use:

for (i in process)
        printf "ldapsearch -p 389 -D userid -w password -h server \"%s
\" -s base \"objectClass=*\" member >>domino-members.txt\n\n", i >
"get-domino-members.cmd"

instead of Vassilis':

for (i in process)
        printf "ldapsearch -p 389 -D userid -w password -h server \"%s
\" -s base \"objectClass=*\" member >>domino-members.txt\n\n",
process[i] > "get-domino-members.cmd"

....keeping in mind that the index itself is the data of interest, not the
throwaway value stored there.

Although I do wonder why all those 'printf()'s don't actually have any
parentheses or format strings. I'd use 'print' myself but maybe your variant
works differently than mine.

- Anton Treuenfels

SP wrote:
<snip>
> The external command "ldapsearch" is given the name of a group and it
> returns a list of members:
> 
> the command:
> ldapsearch ... "CN=Engineering - Stamford Group" ... member
> 
> returns:
> CN=Engineering - Stamford Group
> member=CN=Engineering - Managers
> member=CN=Engineering - Synapse - Product Support
> member=CN=Engineering - Synapse - Software Development
> member=CN=Engineering - Synapse - Support Group
> member=CN=Engineering - Synapse Database
> member=CN=Engineering - Testing
> member=CN=Jane Doe,O=mycompany
> 
> I'm trying to call the command and then process each line of output, I
> found a refernece to
> the |& pipe command, and need an example on how its used.
> 
> Thank you for your feddback.
> 

I'm not sure, but I THINK what you're trying do do is this:

1) You have an input file of "groups", one per line, to be ignored.
2) You have a second file of "groups", one per line, to be processed 
unless they're in that first file.
3) For each "group" that is to be processed, the command
	ldapsearch... "<group>" ... member"
produces output that's a list of details for that group.
4) You want to do some manipulation on that list of details to produce 
your final output.

If that's it, all you need is:

NR == FNR { exclude[$0]++; next }	# first file: exclude list
!exclude[$0] {				# second file: main input
	# ldapsearch ... "CN=Engineering - Stamford Group" ... member
	cmd="ldapsearch ... \"" $0 "\" ... member"
	while ((cmd | getline var) > 0) {
		# do whatever massaging you want to "var" then
		print var
	}
	close(cmd)
}

Please make sure you read and fully understand the implications of using 
getline as discussed at http://tinyurl.com/yn9ka9.

Regards,

	Ed.

On Sep 3, 6:25 am, Ed Morton <mor...@lsupcaemnt.com> wrote:
> SP wrote:
>
> <snip>
>
>
>
> > The external command "ldapsearch" is given the name of a group and it
> > returns a list of members:
>
> > the command:
> > ldapsearch ... "CN=Engineering - Stamford Group" ... member
>
> > returns:
> > CN=Engineering - Stamford Group
> > member=CN=Engineering - Managers
> > member=CN=Engineering - Synapse - Product Support
> > member=CN=Engineering - Synapse - Software Development
> > member=CN=Engineering - Synapse - Support Group
> > member=CN=Engineering - Synapse Database
> > member=CN=Engineering - Testing
> > member=CN=Jane Doe,O=mycompany
>
> > I'm trying to call the command and then process each line of output, I
> > found a refernece to
> > the |& pipe command, and need an example on how its used.
>
> > Thank you for your feddback.
>
> I'm not sure, but I THINK what you're trying do do is this:
>
> 1) You have an input file of "groups", one per line, to be ignored.
> 2) You have a second file of "groups", one per line, to be processed
> unless they're in that first file.
> 3) For each "group" that is to be processed, the command
>         ldapsearch... "<group>" ... member"
> produces output that's a list of details for that group.
> 4) You want to do some manipulation on that list of details to produce
> your final output.
>
> If that's it, all you need is:
>
> NR == FNR { exclude[$0]++; next }       # first file: exclude list
> !exclude[$0] {                          # second file: main input
>         # ldapsearch ... "CN=Engineering - Stamford Group" ... member
>         cmd="ldapsearch ... \"" $0 "\" ... member"
>         while ((cmd | getline var) > 0) {
>                 # do whatever massaging you want to "var" then
>                 print var
>         }
>         close(cmd)
>
> }
>
> Please make sure you read and fully understand the implications of using
> getline as discussed athttp://tinyurl.com/yn9ka9.
>
> Regards,
>
>         Ed.

Thanks to every one for the help.

I was having problems with the "command | getline" portion of the
script, which Ed cleared up for me.
Everyone had good suggestions which I have incorporated into the
script, the script reads a group structure from a Domino LDAP server
and recreates it on a Microsoft Active Directory server.
I needed to automate it because there are thousands of groups and
finding what was changed is extremely time consuming.

Again thanks for all the help.

Sal