I have to search and replace text in an html file without corrupting
the html, can this be done with sed, or anything else?


Ian

Ian Gil <i@NOSPAMALLOWED.com> writes:

> I have to search and replace text in an html file without corrupting
> the html, can this be done with sed, or anything else?

Yes or yes.


-- 
__Pascal_Bourguignon__                          http://www.informatimago.com/
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Living free in Alaska or in Siberia, a grizzli's life expectancy is 35 years,
but no more than 8 years in captivity.           http://www.theadvocates.org/

On Fri, 28 Nov 2003 23:31:56 GMT, Ian Gil <i@NOSPAMALLOWED.com> wrote:
> 
> 
> I have to search and replace text in an html file without corrupting
> the html, can this be done with sed, or anything else?
> 
> 
> Ian
Newsgroups: comp.os.linux.misc
From: Alan Connor <zzzzzz@xxx.yyy>
Subject: Re: sed: ignore html in search and replace?
References: <slrnbsfnio.k4.i@jal.net>
Reply-To: xxxx@yyy.zzz
Followup-To: 

On Fri, 28 Nov 2003 23:31:56 GMT, Ian Gil <i@NOSPAMALLOWED.com> wrote:
> 
> 
> I have to search and replace text in an html file without corrupting
> the html, can this be done with sed, or anything else?

Yes and yes. Sed is great tool. Post a short example of what you want
to do and myself or someone else will show you how it's done.

The basic search and replace script for sed is:

sed 's/search/replace/g' inputfile > outputfile ; mv outputfile inputfile.

Ed, which sed is derived from, is also very effective in scripts or
interactively, modifies the orginal file and is a lot less sensitive
to special characters.

Open the file in ed and type

,s/search/replace/g
w
q

And it's done. In a script it can be done like so:

#!/bin/sh

ed file <<PPP #arbitrary choice of string that won't occur in file
,s/search/replace/g
w
q
PPP

exit 0

I'm using ed right now, as my Usenet pager and editor.

(it lets me read the posts of possible trolls one line at a time,
then if I encounter any abuse I'm outta there, or delete the rest
unseen with +1,$d  and reply once --- they hate it :-)


AC

Yes, that's all I want to do a, a simple search and replace but it will be done
globally and there'll be many of them to do, and since html has html tags and
script areas in it I need to have them ignored.

For example, I might have all instances of 'title' replaced by 'heading'
but if <title> is replaced by <heading> it will corrupt the html.


Ian

On Sat, 29 Nov 2003 01:38:47 GMT, Ian Gil <i@NOSPAMALLOWED.com> wrote:
> 
> 
> Yes, that's all I want to do a, a simple search and replace but it will be done
> globally and there'll be many of them to do, and since html has html tags and
> script areas in it I need to have them ignored.
> 
> For example, I might have all instances of 'title' replaced by 'heading'
> but if <title> is replaced by <heading> it will corrupt the html.

That isn't making any sense. Please post an actual example of exactly what
you want to do. 

No way to deal with this in the general terms that you seem to think it
can be done. Computers are dumb as bricks and need to be told exactly
what to do.

Not going to ask again.

AC

I need this command:

 sed 's/title/heading/g' file.html

to work normally just as you see it there EXCEPT within html tags.

So I don't want "<title>" replaced by "<heading>" because "<title> is an
html tag, and I don't want any html tags changed.



Ian


>> Yes, that's all I want to do a, a simple search and replace but it will be done
>> globally and there'll be many of them to do, and since html has html tags and
>> script areas in it I need to have them ignored.
>> 
>> For example, I might have all instances of 'title' replaced by 'heading'
>> but if <title> is replaced by <heading> it will corrupt the html.
> 
> That isn't making any sense. Please post an actual example of exactly what
> you want to do. 
> 
> No way to deal with this in the general terms that you seem to think it
> can be done. Computers are dumb as bricks and need to be told exactly
> what to do.

On Sat, 29 Nov 2003 02:54:50 GMT, Ian Gil <i@NOSPAMALLOWED.com> wrote:
> 
> 
> I need this command:
> 
>  sed 's/title/heading/g' file.html
> 
> to work normally just as you see it there EXCEPT within html tags.
> 
> So I don't want "<title>" replaced by "<heading>" because "<title> is an
> html tag, and I don't want any html tags changed.
> 
> 
> 
> Ian
> 
> 
>>> Yes, that's all I want to do a, a simple search and replace but it will be done
>>> globally and there'll be many of them to do, and since html has html tags and
>>> script areas in it I need to have them ignored.
>>> 
>>> For example, I might have all instances of 'title' replaced by 'heading'
>>> but if <title> is replaced by <heading> it will corrupt the html.
>> 
>> That isn't making any sense. Please post an actual example of exactly what
>> you want to do. 
>> 
>> No way to deal with this in the general terms that you seem to think it
>> can be done. Computers are dumb as bricks and need to be told exactly
>> what to do.

[ please bottompost ]



PLEASE pay attention! Post an ACTUAL example of the sourcecode you want to
edit, then the same sourcecode that has been edited.

Or go away.

AC

On Sat, 29 Nov 2003 03:58:55 +0000, Alan Connor wrote:

> On Sat, 29 Nov 2003 02:54:50 GMT, Ian Gil <i@NOSPAMALLOWED.com> wrote:

>> I need this command:
>> 
>>  sed 's/title/heading/g' file.html
>> 
>> to work normally just as you see it there EXCEPT within html tags.
>> 
>> So I don't want "<title>" replaced by "<heading>" because "<title> is an
>> html tag, and I don't want any html tags changed.

> PLEASE pay attention! Post an ACTUAL example of the sourcecode you want
> to edit, then the same sourcecode that has been edited.

Are you really so daft that this isn't clear enough for you?  Heck, the
original post was plenty clear for me.

Okay, okay, here's your baby food:

----- begin old.html -----
<html> <head> <title> This is the title of the document </title>
<body> This is the body of the document </body> </html>
------ end old.html ------

----- begin new.html -----
<html> <head> <title> This is the heading of the document </title>
<body> This is the body of the document </body> </html>
------ end new.html ------

Ian Gil <i@NOSPAMALLOWED.com> writes:

> I need this command:
>
>  sed 's/title/heading/g' file.html
>
> to work normally just as you see it there EXCEPT within html tags.
>
> So I don't want "<title>" replaced by "<heading>" because "<title> is an
> html tag, and I don't want any html tags changed.

One approach:

's/title>/saveme>/g'
's/title/heading/g'
's/saveme>/title>/g'

It might be easier to fire up emacs, or some other editor
and use an interactive search/replace.

Ian Gil <i@NOSPAMALLOWED.com> writes:

> Yes, that's all I want to do a, a simple search and replace but it will be done
> globally and there'll be many of them to do, and since html has html tags and
> script areas in it I need to have them ignored.
> 
> For example, I might have all instances of 'title' replaced by 'heading'
> but if <title> is replaced by <heading> it will corrupt the html.

Don't be  silly, any occurence  of "entitled" would be  transformed to
"enheadingd"  then!  When  you want  to run  substitutions  you always
include some context:

   sed -e '/\(prefix context\)\(old\)\(suffix context\)/\1new\3/g'


-- 
__Pascal_Bourguignon__                          http://www.informatimago.com/
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Living free in Alaska or in Siberia, a grizzli's life expectancy is 35 years,
but no more than 8 years in captivity.           http://www.theadvocates.org/

On 29 Nov 2003 05:40:48 +0100, Pascal Bourguignon <spam@thalassa.informatimago.com> wrote:
> 
> 
> Ian Gil <i@NOSPAMALLOWED.com> writes:
> 
>> Yes, that's all I want to do a, a simple search and replace but it will be done
>> globally and there'll be many of them to do, and since html has html tags and
>> script areas in it I need to have them ignored.
>> 
>> For example, I might have all instances of 'title' replaced by 'heading'
>> but if <title> is replaced by <heading> it will corrupt the html.
> 
> Don't be  silly, any occurence  of "entitled" would be  transformed to
> "enheadingd"  then!  When  you want  to run  substitutions  you always
> include some context:
> 
>    sed -n -e '/\(prefix context\)\(old\)\(suffix context\)/\1new\3/g'

Forgot the s :-)

Actually, with the best-case-scenario, these would do the job:

's/abc /def /g'
's/ abc/ def/g'

sed -n -e 's/abc /def /g' -e 's/ abc/ def/g'  inputfile > outputfile
mv outputfile inputfile

But that would be extremely lucky. And if he has a lot of strings needing
replacing, he'll want to feed them to sed -n from a file, rather than writing
all those sed -nscripts.

It's possible that he could isolate the sections without the tags and just
work on them:

sed -n '/RE/,/RE/{s/abc/def/g;p;}' inputfile etc...


AC

Ed Murphy <emurphy42@socal.rr.com> wrote:
> On Sat, 29 Nov 2003 03:58:55 +0000, Alan Connor wrote:

> > On Sat, 29 Nov 2003 02:54:50 GMT, Ian Gil <i@NOSPAMALLOWED.com> wrote:

> >> I need this command:
> >> 
> >>  sed 's/title/heading/g' file.html
> >> 
> >> to work normally just as you see it there EXCEPT within html tags.
> >> 
> >> So I don't want "<title>" replaced by "<heading>" because "<title> is an
> >> html tag, and I don't want any html tags changed.

> > PLEASE pay attention! Post an ACTUAL example of the sourcecode you want
> > to edit, then the same sourcecode that has been edited.

> Are you really so daft that this isn't clear enough for you?  Heck, the
> original post was plenty clear for me.

> Okay, okay, here's your baby food:

> ----- begin old.html -----
> <html> <head> <title> This is the title of the document </title>
> <body> This is the body of the document </body> </html>
> ------ end old.html ------

> ----- begin new.html -----
> <html> <head> <title> This is the heading of the document </title>
> <body> This is the body of the document </body> </html>
> ------ end new.html ------


Then he wants to go for

 sed 's/\([^<]\)title/\1heading/g' file.html
 sed 's/title\([^>]\)/heading\1/g' file.html
 sed 's/title$/heading/g' file.html
 sed 's/^title/heading/g' file.html

Peter

On Sat, 29 Nov 2003 at 01:38 GMT, Ian Gil wrote:
> Yes, that's all I want to do a, a simple search and replace but it will be done
> globally and there'll be many of them to do, and since html has html tags and
> script areas in it I need to have them ignored.
> 
> For example, I might have all instances of 'title' replaced by 'heading'
> but if <title> is replaced by <heading> it will corrupt the html.

     This is not a problem that can easily be solved in a script. It
     is analogous to removing comments from a C source file. There are
     script solutions, but they are hard to understand, and
     consequently not easy to modify when your needs change. Or else
     they are very slow.

     My approach would be to put the tags on lines of their own, since
     whitespace is ignored when rendering HTML, and pipe the result
     through an awk script which would print lines containing tags
     unchanged, and perform substitutions on the other lines.

     I would write a program in C, to first convert all line endings
     to spaces, then insert newlines before "<" and after ">". The
     output would be piped to awk:

htmlfilter < file.html |
   awk  '/^</ { ## print HTML tag and proceed to the next line
                print
                next
              }

         NF   { ## perform substitutions on non-blank lines
                gsub("left","right")
                gsub("title","heading")
                gsub("bold","emphasized")
		## add substitutions here

                ## then print the line
                print
              }' > newfile.html


      A basic filter is quite simple:

/* htmlfilter.c */
#include <stdio.h>
int main (void)
{
  int c;
  int n = 0;
  while ( (c = getchar()) != EOF )
    {
      if ( c == '\n' || c == '\r') c = ' ';
      if ( c == '<' )
	{
	  printf("\n");
	  putchar(c);
	  n = 0;
	}
      else if ( c == '>' )
	{
	  puts( ">" );
	  n = 0;
	}
      else
	{
	  putchar(c);
	}
      ++n;
    }
  putchar('\n');
  return 0;
}
/* EOF */

     This will do for many HTML files, but some elements of a file
     should not be reformatted.

     Dealing with <pre> blocks makes the program more complicated, and
     shows the way to handling any other parts that need special
     attention. This version doesn't put tags on a separate line if
     they occur within a <pre> block.


/* htmlfilter2.c */
#include <stdio.h>

int main (void)
{
  int c;
  int n = 0;
  int pre = 0;
  char string[256]; /* there's no checking for overflow */

  while ( (c = getchar()) != EOF )
    {
      /* convert \n and \r to spaces if not in a <pre> block */
      if ( pre == 0 && (c == '\n' ||  c == '\r') ) c = ' ';

      if ( c == '<' )
        {
          pre = 0;
          n = 0;
        }
      string[n] = c;
      string[n+1] = 0;

      if ( pre == 1 )
        {
          putchar(c);
          if ( !strcmp(string,"</pre") )
            {
              pre = 0;
              n = 0;
              string[n] = '\0';
            }
        }
      else if ( c == '<' )
      /* put a linefeed before the beginning of HTML tag */
        {
          printf("\n%c", c);
          n = 0;
          string[n] = c;
          string[n+1] = 0;
        }
      else if ( c == '>' )
      /* put a linefeed after the end of HTML tag */
        {
          puts( ">" );
          n = 0;
        }
      else if ( c == ' ' )
        {
          if ( n > 74 )
            /* if line is longer than 74 characters, break at a space */
            {
              putchar('\n');
              n = 0;
            }
          else if ( n > 0 )
            {
              putchar(c);
            }
        }
      else
        {
          putchar(c);
        }

      ++n;
      if ( !strcmp(string,"<pre") )
        {
          pre = 1;
          n = 0;
          string[n] = '\0';
        }
    }
  putchar('\n');
  return 0;
}
/* EOF */

    The same thing can be written in a bash script, but it is very
    slow, especially on large files:


#!/bin/bash

print() {
    if [ -n "$line" ]
    then
      if [ $pre -eq 1 ]
      then
	printf "%s\n" "${line%["$NL$CR"]}"
      else
	case $line in
	    '<'*) printf "%s\n" "${line%["$NL$CR"]}" ;;
	    *)  #######  substitutions go here  #######
		line=${line//right/left}
		line=${line//bold/emphasized}
		line=${line//title/heading}
		#######################################
		printf "%s\n" "${line%["$NL$CR"]}"
		;;
	esac
	line=
      fi
    fi
}

NL=$'\n'
CR=$'\r'
line=
pre=0
n=0

while IFS= read -d '' -rn1 c
do
  case $line in
      "<pre") pre=1 ;;
      "</pre") pre=0 ;;
  esac
  if [ $pre -eq 1 ]
  then
    line=$line$c
    case $c in
	"$NL")
	    print
	    line= ;;
    esac
  else
    line=${line# }
    case $c in
	"$NL"|"$CR") line="${line% } " ;;
	'<') print
	    line=$c
	    ;;
	'>') line=$line$c 
	    print
	    line=
	    ;;
	' ')
	    if [ ${#line} -gt 74 ]
	    then
	      print
	      line=
	    else
	      [ -n "$line" ] && line="${line% } "
	    fi
	    ;;
	*) line="${line}${c}" ;;
    esac
  fi
done
print
## EOF ##


      These have not been thoroughly tested, but they may provide a
      base on which to work.

-- 
    Chris F.A. Johnson                        http://cfaj.freeshell.org
    ===================================================================
    My code (if any) in this post is copyright 2003, Chris F.A. Johnson
    and may be copied under the terms of the GNU General Public License

>> For example, I might have all instances of 'title' replaced by 'heading'
>> but if <title> is replaced by <heading> it will corrupt the html.
> 
>      This is not a problem that can easily be solved in a script...

I figured things might start to get hairy...and you're just the barber I
was looking for - thanks!

Just one thing about the awk part, I'd like to only replace complete
words; so in the other scripts I was able to sandwich the words between
the regex \b to match only complete words but I wasn't able to do that
in this awk line:

                 gsub("title","heading")


[..in other posts..]

This appeared to work
  >sed 's/\btitle\b\([^>]\)/heading\1/g' file.html |
  >sed 's/title$/heading/g' |
  >sed 's/^title/heading/g' 
but was this 
  >sed -e 's/\(prefix context\)\(old\)\(suffix context\)/\1new\3/g'
supposed to do the same thing? 
If so I wasn't able make it work. I'd used "<" and ">" in place of 
"prefix context" and "suffix context".


Thanks,

Ian

"IG" == Ian Gil <i@NOSPAMALLOWED.com>:
IG> Yes, that's all I want to do a, a simple search and replace but it will be done
IG> globally and there'll be many of them to do, and since html has html tags and
IG> script areas in it I need to have them ignored.
IG> 
IG> For example, I might have all instances of 'title' replaced by 'heading'
IG> but if <title> is replaced by <heading> it will corrupt the html.

The HTML tags can be lowercase, uppercase, or mixed case. 'sed' is not
very well equipped to handle changes while ignoring case. Use 'awk' or
'perl' instead.

-=-
This message was posted via two or more anonymous remailing services.

"IG" == Ian Gil <i@NOSPAMALLOWED.com>:
IG> Just one thing about the awk part, I'd like to only replace complete
IG> words; so in the other scripts I was able to sandwich the words between
IG> the regex \b to match only complete words but I wasn't able to do that
IG> in this awk line:
IG>                  gsub("title","heading")
IG> If so I wasn't able make it work. I'd used "<" and ">" in place of 
IG> "prefix context" and "suffix context".

You should have used '\\<' and '\\>' instead. E.g.:

$ seq 20|awk '{gsub("\\<2","#");print $0}'
1
#
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
#0

-=-
This message was posted via two or more anonymous remailing services.

On Sat, 29 Nov 2003 at 12:24 GMT, Ian Gil wrote:
>>> For example, I might have all instances of 'title' replaced by 'heading'
>>> but if <title> is replaced by <heading> it will corrupt the html.
>> 
>>      This is not a problem that can easily be solved in a script...
> 
> I figured things might start to get hairy...and you're just the barber I
> was looking for - thanks!
> 
> Just one thing about the awk part, I'd like to only replace complete
> words; so in the other scripts I was able to sandwich the words between
> the regex \b to match only complete words but I wasn't able to do that
> in this awk line:
> 
>                  gsub("title","heading")

gsub(/\<title\>/,"heading")


-- 
    Chris F.A. Johnson                        http://cfaj.freeshell.org
    ===================================================================
    My code (if any) in this post is copyright 2003, Chris F.A. Johnson
    and may be copied under the terms of the GNU General Public License

On Sat, 29 Nov 2003 12:24:14 GMT, Ian Gil <i@NOSPAMALLOWED.com> wrote:
> 
> 
>>> For example, I might have all instances of 'title' replaced by 'heading'
>>> but if <title> is replaced by <heading> it will corrupt the html.
>> 
>>      This is not a problem that can easily be solved in a script...
> 
> I figured things might start to get hairy...and you're just the barber I
> was looking for - thanks!
> 
> Just one thing about the awk part, I'd like to only replace complete
> words; so in the other scripts I was able to sandwich the words between
> the regex \b to match only complete words but I wasn't able to do that
> in this awk line:
> 
>                  gsub("title","heading")
> 
> 
> [..in other posts..]
> 
> This appeared to work
>  >sed 's/\btitle\b\([^>]\)/heading\1/g' file.html |
>  >sed 's/title$/heading/g' |
>  >sed 's/^title/heading/g' 
> but was this 
>  >sed -e 's/\(prefix context\)\(old\)\(suffix context\)/\1new\3/g'
> supposed to do the same thing? 
> If so I wasn't able make it work. I'd used "<" and ">" in place of 
> "prefix context" and "suffix context".
> 
> 
> Thanks,
> 
> Ian

Killfiled for 90 days 

People: If he won't post a sample of the HTML source is working with,
then he is up to no good. 

You are probably aiding and abetting a script-kiddie.

AC

On Sat, 29 Nov 2003 19:32:20 GMT, Alan Connor <zzzzzz@xxx.yyy> wrote:
> 
> 
> On Sat, 29 Nov 2003 12:24:14 GMT, Ian Gil <i@NOSPAMALLOWED.com> wrote:
>> 
>> 
>>>> For example, I might have all instances of 'title' replaced by 'heading'
>>>> but if <title> is replaced by <heading> it will corrupt the html.
>>> 
>>>      This is not a problem that can easily be solved in a script...
>> 
>> I figured things might start to get hairy...and you're just the barber I
>> was looking for - thanks!
>> 
>> Just one thing about the awk part, I'd like to only replace complete
>> words; so in the other scripts I was able to sandwich the words between
>> the regex \b to match only complete words but I wasn't able to do that
>> in this awk line:
>> 
>>                  gsub("title","heading")
>> 
>> 
>> [..in other posts..]
>> 
>> This appeared to work
>>  >sed 's/\btitle\b\([^>]\)/heading\1/g' file.html |
>>  >sed 's/title$/heading/g' |
>>  >sed 's/^title/heading/g' 
>> but was this 
>>  >sed -e 's/\(prefix context\)\(old\)\(suffix context\)/\1new\3/g'
>> supposed to do the same thing? 
>> If so I wasn't able make it work. I'd used "<" and ">" in place of 
>> "prefix context" and "suffix context".
>> 
>> 
>> Thanks,
>> 
>> Ian
> 
> Killfiled for 90 days 
> 
> People: If he won't post a sample of the HTML source is working with,
> then he is up to no good. 
> 
> You are probably aiding and abetting a script-kiddie.
> 
> AC
> 

Go it: This person is probably stealing the HTML source that someone
worked their butt off to create.

THAT'S why he won't post it.

AC

You're a strange bloke AC, but I don't like to hear you
cry so here you are; it is not any one particular html, if that
were the case I'd simply edit the source with an editor. The source
will be german web pages which will translate and highlight key words
on the fly, for my own viewing, on my browser, via lua extensions
which allow that sort of thing - pretty cool eh - it bascially allows
you to manipulate your web browsing on the fly. So do a search for
'das baby' at google.de (pun intended) and digest that html as the
baby food you so require.

On Sat, 29 Nov 2003 22:24:32 +0000, Ian Gil wrote:

> You're a strange bloke AC, but I don't like to hear you
> cry so here you are; it is not any one particular html, if that
> were the case I'd simply edit the source with an editor. The source
> will be german web pages which will translate and highlight key words
> on the fly, for my own viewing, on my browser, via lua extensions
> which allow that sort of thing - pretty cool eh - it bascially allows
> you to manipulate your web browsing on the fly. So do a search for
> 'das baby' at google.de (pun intended) and digest that html as the
> baby food you so require.

If the pages are in German, but the HTML tags are (as usual) in English,
then how often will the text being replaced actually appear within a
tag?  (Well, I suppose URLs within tags are reasonably at risk.)

> If the pages are in German, but the HTML tags are (as usual) in English,
> then how often will the text being replaced actually appear within a
> tag?  (Well, I suppose URLs within tags are reasonably at risk.)

I don't really know which is why I'm weeding out the tags including java
scripts, etc. Also, some web pages are lengthy tables with acroyms whose
key words can crop up in just about any language - natural or
artificial. All it takes is one character to render a whole web page
incomprehensible.

Ian