proj home

Files   Classes   Functions   Hierarchy  

tokenizerlocal Class Reference

Primitive tag parser, supporting reading and writing. The parser generally looks at the local scope. More...

#include <tokenizerlocal.h>

Inheritance diagram for tokenizerlocal:
Collaboration diagram for tokenizerlocal:

List of all members.

Public Member Functions

stringc debug01 () const
 Print out what i1 i2 iend point to.
 tokenizerlocal (tokenizer &ref_)
 Constructor.
 tokenizerlocal (tokenizerlocal &tl)
 Scope this to [tl.i1,tl.i2].
boolc atomize_next (stringc &atom1, stringc &atom2)
void reset ()
 current and i1 point to start of list.
void reset (tokenizerlocal &tkl)
 Reset ot tkl's scope.
void endpointcontract ()
 iend=i2
void endpointexpand ()
 i2=iend
void endpointreset ()
 iend set to end of list.
boolc operator! () const
 Iterate between [i1,i2).
stringc & operator() () const
 Access the current token in the stream.
void operator++ ()
 Increment the stream's index.
boolc readtag (string &val, stringc &tag)
 Assumes i1 points to atomized tag, i2 to the tag end.
boolc writetag (stringc &val, stringc &tag)
 Searches from current to find, erase and insert string.
boolc erasetag (stringc &tag)
 Assumes i1 points to atomized tag, i2 to the tag end.
boolc scopesearch (stringc &tag)
 Assumes atomized tags search for matching tags: [i1,i2] - [<tag>,</tag>] and current points to i1.
boolc read (string &str, stringc &tag1)
boolc read (string &str, stringc &tag1, stringc &tag2)
boolc read (string &str, stringc &tag1, stringc &tag2, stringc &tag3)
boolc read (string &str, stringc &tag1, stringc &tag2, stringc &tag3, stringc &tag4)
boolc read (string &str, stringc &tag1, stringc &tag2, stringc &tag3, stringc &tag4, stringc &tag5)
boolc read (string &str, stringc &tag1, stringc &tag2, stringc &tag3, stringc &tag4, stringc &tag5, stringc &tag6)
boolc boundscope (stringc &tag1)
 Set iend=i2 after scoping.
boolc scope (stringc &tag1)
boolc scope (stringc &tag1, stringc &tag2)
boolc scope (stringc &tag1, stringc &tag2, stringc &tag3)
boolc scope (stringc &tag1, stringc &tag2, stringc &tag3, stringc &tag4)
boolc scope (stringc &tag1, stringc &tag2, stringc &tag3, stringc &tag4, stringc &tag5)
boolc scope (stringc &tag1, stringc &tag2, stringc &tag3, stringc &tag4, stringc &tag5, stringc &tag6)

Public Attributes

tokenizer & ref
 Reference.
tokenizerlocal * parent
 If no parent global scope.
liststringi i1
 Point to first tag or before.
liststringi i2
 Point to second.
liststringi iend
 End boundary of linear search.

Detailed Description

Primitive tag parser, supporting reading and writing. The parser generally looks at the local scope.

Attributes <sometag hell="true"> not supported. However they could be pre-processed into strict tags.

Use scope() to find matching tags once they have been atomized. Combinations of scope and others are then possible. e.g. scope and erasetag.

atomize(tag) create scoped tags. The scope search can be bounded by setting iend. e.g. Search in a scoped region for a sub scope.

Definition at line 21 of file tokenizerlocal.h.

Constructor & Destructor Documentation

tokenizerlocal::tokenizerlocal ( tokenizer &  ref_  ) 

Constructor.

Definition at line 6 of file tokenizerlocal.cpp.

References tokenizer::current, i1, i2, iend, ref, and tokenizer::seq.

00007   : ref(ref_), parent(0)
00008 {
00009   i1=ref.current;
00010   i2=ref.seq.end();
00011   iend=ref.seq.end();
00012 }

tokenizerlocal::tokenizerlocal ( tokenizerlocal &  tl  ) 

Scope this to [tl.i1,tl.i2].

Definition at line 15 of file tokenizerlocal.cpp.

References reset().

00018   : ref(tl.ref), parent(&tl)
00019 {
00020   reset(tl);
00021 }

Member Function Documentation

boolc tokenizerlocal::atomize_next ( stringc &  atom1,
stringc &  atom2 
)

Definition at line 35 of file tokenizerlocal.cpp.

00039 {
00040   if (ref.atomize_next(atom1)==false)
00041     return false;
00042   i1=ref.current;
00043 
00044   if (ref.atomize_next(atom2)==false)
00045     return false;
00046   i2=ref.current;
00047 
00048   return true;
00049 }

boolc tokenizerlocal::boundscope ( stringc &  tag1  ) 

Set iend=i2 after scoping.

Definition at line 333 of file tokenizerlocal.cpp.

References i2, iend, and scope().

Referenced by modulereport::unittestshtmldetailed().

00334 {
00335   if (scope(tag1)==false)
00336     return false;
00337 
00338   iend=i2;
00339   return true;
00340 }

stringc tokenizerlocal::debug01 (  )  const

Print out what i1 i2 iend point to.

Definition at line 375 of file tokenizerlocal.cpp.

References i1, i2, and iend.

Referenced by misclib_testcode::tokenizertest::unittest04().

00376 {
00377   string s0;
00378   s0 += ("i1=" + *i1);
00379   s0 += (" i2=" + *i2);
00380   s0 += (" iend=" + *iend);
00381   return s0;
00382 }

void tokenizerlocal::endpointcontract (  ) 

iend=i2

Definition at line 169 of file tokenizerlocal.cpp.

References i2, and iend.

00170 {
00171   iend=i2;
00172 }

void tokenizerlocal::endpointexpand (  ) 

i2=iend

Definition at line 174 of file tokenizerlocal.cpp.

References i2, and iend.

00175 {
00176   i2=iend;
00177 }

void tokenizerlocal::endpointreset (  ) 

iend set to end of list.

Definition at line 179 of file tokenizerlocal.cpp.

References i2, iend, ref, and tokenizer::seq.

00180 {
00181   iend=ref.seq.end();
00182   i2=iend;
00183 }

boolc tokenizerlocal::erasetag ( stringc &  tag  ) 

Assumes i1 points to atomized tag, i2 to the tag end.

Definition at line 225 of file tokenizerlocal.cpp.

References assertreturnfalse, tokenizer::current, i1, i2, ref, and tokenizer::seq.

Referenced by misclib_testcode::tokenizertest::unittest02().

00226 {
00227   ref.current=i1;
00228   assertreturnfalse( ref()==("<"+tag+">"));
00229   assertreturnfalse( *i2==("</"+tag+">") );
00230   ++i2;
00231   ref.seq.erase(i1,i2);
00232 
00233   return true;
00234 }

boolc tokenizerlocal::operator! (  )  const

Iterate between [i1,i2).

Definition at line 134 of file tokenizerlocal.cpp.

References assertreturnfalse, tokenizer::current, i2, iend, and ref.

00135 { 
00136   if (ref.current==i2) 
00137     return false; 
00138   assertreturnfalse(ref.current!=iend); 
00139   return true; 
00140 }

stringc & tokenizerlocal::operator() (  )  const

Access the current token in the stream.

Definition at line 23 of file tokenizerlocal.cpp.

References tokenizer::current, and ref.

00024 { 
00025   return *ref.current; 
00026 }

void tokenizerlocal::operator++ (  ) 

Increment the stream's index.

Definition at line 28 of file tokenizerlocal.cpp.

References tokenizer::current, i2, and ref.

00029 { 
00030   if (ref.current!=i2) 
00031     ++ref; 
00032 }

boolc tokenizerlocal::read ( string &  str,
stringc &  tag1,
stringc &  tag2,
stringc &  tag3,
stringc &  tag4,
stringc &  tag5,
stringc &  tag6 
)

Definition at line 325 of file tokenizerlocal.cpp.

References readtag(), and scope().

00326 {
00327   if( scope(tag1,tag2,tag3,tag4,tag5,tag6)==false)
00328     return false;
00329 
00330   return readtag(str,tag6);
00331 }

boolc tokenizerlocal::read ( string &  str,
stringc &  tag1,
stringc &  tag2,
stringc &  tag3,
stringc &  tag4,
stringc &  tag5 
)

Definition at line 317 of file tokenizerlocal.cpp.

References readtag(), and scope().

00318 {
00319   if( scope(tag1,tag2,tag3,tag4,tag5)==false)
00320     return false;
00321 
00322   return readtag(str,tag5);
00323 }

boolc tokenizerlocal::read ( string &  str,
stringc &  tag1,
stringc &  tag2,
stringc &  tag3,
stringc &  tag4 
)

Definition at line 309 of file tokenizerlocal.cpp.

References readtag(), and scope().

00310 {
00311   if( scope(tag1,tag2,tag3,tag4)==false)
00312     return false;
00313 
00314   return readtag(str,tag4);
00315 }

boolc tokenizerlocal::read ( string &  str,
stringc &  tag1,
stringc &  tag2,
stringc &  tag3 
)

Definition at line 301 of file tokenizerlocal.cpp.

References readtag(), and scope().

00302 {
00303   if( scope(tag1,tag2,tag3)==false)
00304     return false;
00305 
00306   return readtag(str,tag3);
00307 }

boolc tokenizerlocal::read ( string &  str,
stringc &  tag1,
stringc &  tag2 
)

Definition at line 293 of file tokenizerlocal.cpp.

References readtag(), and scope().

00294 {
00295   if( scope(tag1,tag2)==false)
00296     return false;
00297 
00298   return readtag(str,tag2);
00299 }

boolc tokenizerlocal::read ( string &  str,
stringc &  tag1 
)

Definition at line 285 of file tokenizerlocal.cpp.

References readtag(), and scope().

Referenced by modulereport::compile_exitstatus(), modulereport::compilerhtmldetailed(), html::insert(), mkerrorsconfig::read(), misclib_testcode::tokenizertest::unittest02(), misclib_testcode::tokenizertest::unittest03(), misclib_testcode::tokenizertest::unittest04(), modulereport::unittests_exitstatus(), and modulereport::unittestshtmldetailed().

00286 {
00287   if( scope(tag1)==false)
00288     return false;
00289 
00290   return readtag(str,tag1);
00291 }

boolc tokenizerlocal::readtag ( string &  val,
stringc &  tag 
)

Assumes i1 points to atomized tag, i2 to the tag end.

Definition at line 261 of file tokenizerlocal.cpp.

References assertreturnfalse, and ref.

Referenced by read().

00262 {
00263   val="";
00264   assertreturnfalse( ref()==("<"+tag+">"));
00265   ++ref;
00266 
00267   static long int counter;
00268   counter=0;
00269 
00270   stringc tagend("</"+tag+">");
00271   val.empty();
00272   for ( ; ref()!=tagend; ++ref )
00273   {
00274     val += ref();
00275 
00276     // Exit counter logic if not end token found.
00277     ++counter;
00278     if(counter>countermax)
00279       return false;
00280   }
00281 
00282   return true;
00283 }

void tokenizerlocal::reset ( tokenizerlocal &  tkl  ) 

Reset ot tkl's scope.

Definition at line 161 of file tokenizerlocal.cpp.

References tokenizer::current, i1, i2, iend, and ref.

00162 {
00163   i1=tl.i1;
00164   i2=tl.i2;
00165   iend=tl.i2;
00166   ref.current=i1;
00167 }

void tokenizerlocal::reset (  ) 

current and i1 point to start of list.

Definition at line 142 of file tokenizerlocal.cpp.

References tokenizer::current, i1, i2, iend, parent, ref, and tokenizer::reset().

Referenced by modulereport::compilerhtmldetailed(), mkerrorsconfig::read(), misclib_testcode::tokenizertest::unittest02(), misclib_testcode::tokenizertest::unittest03(), and misclib_testcode::tokenizertest::unittest04().

00143 {
00144   // Global scope. No parent so reset the tokenizer.
00145   if (parent==0)
00146   {
00147     ref.reset();
00148     i1 = ref.current;
00149 
00150     return;
00151   }
00152 
00153   // parent != 0
00154   // Assume valid pointer.
00155   i1=parent->i1;
00156   i2=parent->i2;
00157   iend=parent->i2;
00158   ref.current=parent->i1;
00159 }

boolc tokenizerlocal::scope ( stringc &  tag1,
stringc &  tag2,
stringc &  tag3,
stringc &  tag4,
stringc &  tag5,
stringc &  tag6 
)

Definition at line 119 of file tokenizerlocal.cpp.

00127 {
00128   if (scope(tag1,tag2,tag3,tag4,tag5)==false)
00129     return false;
00130 
00131   return scope(tag6);
00132 }

boolc tokenizerlocal::scope ( stringc &  tag1,
stringc &  tag2,
stringc &  tag3,
stringc &  tag4,
stringc &  tag5 
)

Definition at line 104 of file tokenizerlocal.cpp.

00111 {
00112   if (scope(tag1,tag2,tag3,tag4)==false)
00113     return false;
00114 
00115   return scope(tag5);
00116 }

boolc tokenizerlocal::scope ( stringc &  tag1,
stringc &  tag2,
stringc &  tag3,
stringc &  tag4 
)

Definition at line 90 of file tokenizerlocal.cpp.

00096 {
00097   if (scope(tag1,tag2,tag3)==false)
00098     return false;
00099 
00100   return scope(tag4);
00101 }

boolc tokenizerlocal::scope ( stringc &  tag1,
stringc &  tag2,
stringc &  tag3 
)

Definition at line 77 of file tokenizerlocal.cpp.

00082 {
00083   if (scope(tag1,tag2)==false)
00084     return false;
00085 
00086   return scope(tag3);
00087 }

boolc tokenizerlocal::scope ( stringc &  tag1,
stringc &  tag2 
)

Definition at line 64 of file tokenizerlocal.cpp.

00068 {
00069   if (scope(tag1)==false)
00070     return false;
00071 
00072 //cout << "found " << SHOW(tag1) << endl;
00073   return scope(tag2);
00074 }

boolc tokenizerlocal::scope ( stringc &  tag1  ) 

Definition at line 51 of file tokenizerlocal.cpp.

References tokenizer::atomize_next_tag(), i1, i2, iend, and ref.

Referenced by boundscope(), tokenizerlocalvar::operator!(), read(), mkerrorsconfig::read(), misclib_testcode::tokenizertest::unittest01(), misclib_testcode::tokenizertest::unittest02(), misclib_testcode::tokenizertest::unittest03(), misclib_testcode::tokenizertest::unittest04(), and modulereport::unittestshtmldetailed().

00052 { 
00053   //bool res = ref.atomize_next_tag(i1,i2,tag1); 
00054   bool res = ref.atomize_next_tag(i1,i2,tag1,iend); 
00055 
00056 
00057 
00058 //cout << "scope(stringc&) " << res << endl;
00059 //cout << "  " << SHOW(*i1) << " " << SHOW(*i2) << SHOW(ref.current) << endl;
00060   return res;
00061 }

boolc tokenizerlocal::scopesearch ( stringc &  tag  ) 

Assumes atomized tags search for matching tags: [i1,i2] - [<tag>,</tag>] and current points to i1.

Definition at line 185 of file tokenizerlocal.cpp.

References tokenizer::current, i1, i2, iend, and ref.

Referenced by misclib_testcode::tokenizertest::unittest02(), and writetag().

00186 {
00187   // Using the current as a starting point.
00188   string tag1=("<"+tag+">");
00189 
00190   for ( ; !ref; )
00191   {
00192     if (*ref==tag1)
00193       break;
00194 
00195     if (ref.current!=iend)
00196       ++ref;
00197   }
00198   if (ref.current==iend)
00199     return false;
00200 
00201   i1 = ref.current;
00202   ++ref;
00203 
00204   string tag2=("</"+tag+">");
00205   for ( ; !ref; )
00206   {
00207     if (*ref==tag2)
00208       break;
00209     if (ref.current!=iend)
00210       ++ref;
00211   }
00212   if (ref.current==iend)
00213     return false;
00214 
00215   i2 = ref.current;
00216 
00217   ref.current=i1;
00218 
00219   assert(*i1==tag1);
00220   assert(*i2==tag2);
00221 
00222   return true;
00223 }

boolc tokenizerlocal::writetag ( stringc &  val,
stringc &  tag 
)

Searches from current to find, erase and insert string.

Definition at line 236 of file tokenizerlocal.cpp.

References assertreturnfalse, tokenizer::current, i1, i2, ref, scopesearch(), and tokenizer::seq.

Referenced by misclib_testcode::tokenizertest::unittest02(), and misclib_testcode::tokenizertest::unittest03().

00237 {
00238   //ref.current=i1;
00239 
00240   assertreturnfalse( scopesearch(tag) );
00241 
00242   assert( *i1==("<"+tag+">"));
00243   ++ref;  
00244   assert( *i2==("</"+tag+">") );
00245 /*
00246   assertreturnfalse( ref()==("<"+tag+">"));
00247   ++ref;  
00248   assertreturnfalse( *i2==("</"+tag+">") );
00249 */
00250   
00251   ref.seq.erase(ref.current,i2);
00252 
00253   // Insert the new element in.
00254   liststringi k=i1;
00255   ++k;
00256   ref.seq.insert(k,val);
00257   
00258   return true;
00259 }

Member Data Documentation

liststringi tokenizerlocal::i1

Point to first tag or before.

Definition at line 34 of file tokenizerlocal.h.

Referenced by debug01(), erasetag(), reset(), scope(), scopesearch(), tokenizerlocal(), misclib_testcode::tokenizertest::unittest02(), and writetag().

liststringi tokenizerlocal::i2

Point to second.

Definition at line 36 of file tokenizerlocal.h.

Referenced by boundscope(), debug01(), endpointcontract(), endpointexpand(), endpointreset(), erasetag(), operator!(), operator++(), reset(), scope(), scopesearch(), tokenizerlocal(), misclib_testcode::tokenizertest::unittest02(), and writetag().

liststringi tokenizerlocal::iend

End boundary of linear search.

Definition at line 38 of file tokenizerlocal.h.

Referenced by boundscope(), debug01(), endpointcontract(), endpointexpand(), endpointreset(), operator!(), reset(), scope(), scopesearch(), and tokenizerlocal().

tokenizerlocal* tokenizerlocal::parent

If no parent global scope.

Definition at line 31 of file tokenizerlocal.h.

Referenced by reset().

tokenizer& tokenizerlocal::ref

Reference.

Definition at line 28 of file tokenizerlocal.h.

Referenced by endpointreset(), erasetag(), operator!(), operator()(), operator++(), readtag(), reset(), scope(), scopesearch(), tokenizerlocal(), misclib_testcode::tokenizertest::unittest02(), misclib_testcode::tokenizertest::unittest03(), and writetag().

The documentation for this class was generated from the following files:
Generated on Fri Mar 4 00:50:20 2011 for Chelton Evans Source by  doxygen 1.5.8