tokenizer Class Reference

Primitive parser with state. More...

#include <tokenizer.h>

Collaboration diagram for tokenizer:

List of all members.

Public Member Functions
void	reset ()
	Resets token iteration.
boolc	operator! () const
	Can the stream be read?
string &	operator* ()
	Modify the current token.
stringc &	operator() () const
	Access the current token in the stream.
void	operator++ ()
	Increment the stream's index.
void	atomize (stringc &atom)
	Splits strings.
void	subtract (stringc &atom)
	Splits strings removing the atom.
void	tokenize ()
	delimiter set.
void	stripcomment (stringc &comment)
	For each line delete right of and including the comment if it exists in the line.
void	remove (stringc &token)
	Remove matching tokens.
void	remove_if ()
	Remove invalid lines as interpreted by myspacer.
template<typename SPACER >
void	remove_if (SPACER spacer)
	Remove invalid lines.
template<typename X >
void	apply (X x)
	Apply the functional object x to each element.
void	trim ()
	Trim the leading and trailing space of each token.
void	trim_and_prune ()
	Remove surrounding white space and delete empty strings.
void	extractfromcurrent (vector< string > &v, stringc &atom) const
	The current line is read and parsed, split by the atom and each element is written to the vector v.
ostream &	print (ostream &os) const
	Print seq.
void	read (stringc &data)
	Read the data in.
void	readaslines (stringc &data)
	Read the string as lines.
void	readaslinesgeneral (stringc &data)
	Because special characters in strings can interfere with reading strings, this routine is a hack to remove white spaces, commas and empty lines.
	tokenizer ()
	Construct in uninitialized state.
	tokenizer (stringc &data)
	Construct with an initial string.
boolc	operator== (tokenizer &t2)
	Data copy.
boolc	find (string::size_type &k, stringc &atom)
	Non destructive text search.
boolc	find (string::size_type &k, stringc &atom, string::size_type const k0)
	Search from k0 in the current string.
boolc	atomize_next (stringc &atom)
	Search for the first atom and atomize it.
boolc	atomize_next (stringc &atom, liststringi &iend_)
boolc	atomize_next_tag (liststringi &i1, liststringi &i2, stringc &tag, liststringi &iend_)
	tag=cat then i1 points to <cat>, i2 to </cat>.
boolc	atomize_next_tag (liststringi &i1, liststringi &i2, stringc &tag)
boolc	atomize_next (liststringi &i1, liststringi &i2, stringc &atom1, stringc &atom2)
	operator stringc ()
	Output the tokenizer back as a string.
Public Attributes
liststringi	current
	The current state.
list< string >	seq
	Data representation.
string	printdelimiter
	Each element printed is separated by the delimiter.

---

Detailed Description

Primitive parser with state.

Generally concerned with global scope.

This class has two roles. Firstly it is a primitive parser. Secondly it has a state so this class is used by another to point to the current token.

I refer to a stream to mean the list of string tokens with a current position in the list.

Use: read in the file and call atomize to separate the sequence into individual elements that can be interpreted by an interpreter.

Can directly operate on the data though the list class.

Example 1. Remove comments and delete any empty strings.

  tokenizer ss;
  ...
  ss.stripcomment("//");
  ss.seq.remove("");

See vrmlshape where I wrote a VRML parser using this class to hold the current state.

Definition at line 66 of file tokenizer.h.

---

Constructor & Destructor Documentation

tokenizer::tokenizer

(

)

Construct in uninitialized state.

Definition at line 435 of file tokenizer.cpp.

References current, and seq.

  : printdelimiter("\n")
{
  current=seq.end();
}

tokenizer::tokenizer

(

stringc &

data

)

Construct with an initial string.

Definition at line 441 of file tokenizer.cpp.

References reset(), and seq.

{
  seq.push_back(data);
  reset();
}

---

Member Function Documentation

template<typename X >

void tokenizer::apply

(

X

x

)

[inline]

Apply the functional object x to each element.

Definition at line 117 of file tokenizer.h.

References seq.

Referenced by trim().

  { 
    liststringi i = seq.begin();
    liststringic imax = seq.end();
    for ( ; i!=imax; ++i )
      { x(*i); }
  };

void tokenizer::atomize

(

stringc &

atom

)

Splits strings.

Definition at line 297 of file tokenizer.cpp.

References seq.

{
  liststringi i = seq.begin();

  for (;i!=seq.end(); ++i)
    atomize(i,atom);
}

boolc tokenizer::atomize_next	(	liststringi &	i1,
		liststringi &	i2,
		stringc &	atom1,
		stringc &	atom2
	)

Definition at line 229 of file tokenizer.cpp.

{
  bool res;
  
  res=atomize_next(atom1);
  if (res==false)
    return false;
  i1=current;

  res=atomize_next(atom2);
  if (res==false)
    return false;
  i2=current;

  // Default to reset iterator to first tag.
  current=i1;

  return true;
}

boolc tokenizer::atomize_next	(	stringc &	atom,
		liststringi &	iend_
	)

Definition at line 186 of file tokenizer.cpp.

{
  liststringi i = current;

  string::size_type k;
  
  string::size_type const atomlen = atom.length();

  for (;i!=iend_; ++i)
  {
    k = i->find(atom.c_str());

    if (k==string::npos)
      continue;

    atomize(i,atom,k);
    if (i->length()==atomlen)
    {
      if (atom==*i)
      {
        current=i;
        return true;
      }
    }

    ++i;
    atomize(i,atom,0);
    assert(i->length()==atomlen);
    current=i;
    return true;
  }

  return false; 
}

boolc tokenizer::atomize_next

(

stringc &

atom

)

Search for the first atom and atomize it.

current points to it. Essentially this is a find function.

Definition at line 142 of file tokenizer.cpp.

Referenced by misclib_testcode::tokenizertest::test10().

{
  liststringi i = current;

  string::size_type k;
  
  string::size_type const atomlen = atom.length();

  for (;i!=seq.end(); ++i)
  {
    k = i->find(atom.c_str());

    if (k==string::npos)
      continue;

    atomize(i,atom,k);
    if (i->length()==atomlen)
    {
      if (atom==*i)
      {
        current=i;
        return true;
      }
    }

    ++i;
    atomize(i,atom,0);
    assert(i->length()==atomlen);
    current=i;
    return true;
  }

  return false; 
}

boolc tokenizer::atomize_next_tag	(	liststringi &	i1,
		liststringi &	i2,
		stringc &	tag
	)

Definition at line 255 of file tokenizer.cpp.

{
  liststringi iend_=seq.end();
  return atomize_next_tag(i1,i2,tag,iend_);
}

boolc tokenizer::atomize_next_tag	(	liststringi &	i1,
		liststringi &	i2,
		stringc &	tag,
		liststringi &	iend_
	)

tag=cat then i1 points to <cat>, i2 to </cat>.

Definition at line 266 of file tokenizer.cpp.

Referenced by tokenizerlocal::scope(), and misclib_testcode::tokenizertest::unittest01().

{
//cout << "atomize_next_tag ";
  string tag1="<"+tag+">";
//cout << SHOW(tag1) << " ";
  bool res;

  res=atomize_next(tag1,iend_);
  if (res==false)
    return false;
  i1=current;

  string tag2="</"+tag+">";
//cout << SHOW(tag2) << " ";
  res=atomize_next(tag2,iend_);
  if (res==false)
    return false;
  i2=current;

  // Default to reset iterator to first tag.
  current=i1;
// cout << SHOW(*current) << " 1" << endl;

  return true;
}

void tokenizer::extractfromcurrent	(	vector< string > &	v,
		stringc &	atom
	)			const

The current line is read and parsed, split by the atom and each element is written to the vector v.

Definition at line 306 of file tokenizer.cpp.

{
  v.clear();

  string s(*current);

  if (s.empty())
    return;

  string::size_type k;

  string::size_type const atomlen = atom.length();

  k = s.find(atom.c_str());
  for ( ; k!=string::npos; k=s.find(atom.c_str()) )
  {
    if (k==0)
    {
      s.erase(k,atomlen);

      continue;
    }

    v.push_back(s.substr(0,k));
    s.erase(0,k);
  }

  if (s.empty()==false)
    v.push_back(s);
}

boolc tokenizer::find	(	string::size_type &	k,
		stringc &	atom,
		string::size_type const	k0
	)

Search from k0 in the current string.

Definition at line 503 of file tokenizer.cpp.

References assertreturnfalse, and find().

{
  string::size_type atomsize=atom.size();
  if (atomsize==0)
  {
    current=seq.end();
    return false;
  }

  liststringi i = current;
  assertreturnfalse(i!=seq.end());

  if (k0+atomsize-1<i->size())
  {
    k = i->find(atom.c_str(),k0);
    if (k!=string::npos)
      return true;
  }
    
  // failed to find in current string.
  ++current;

  return tokenizer::find(k,atom);
}

boolc tokenizer::find	(	string::size_type &	k,
		stringc &	atom
	)

Non destructive text search.

Definition at line 533 of file tokenizer.cpp.

Referenced by find(), tokenizerfind::operator++(), tokenizerfind::reset(), and misclib_testcode::tokenizertest::unittest01().

{
  liststringi i = current;

  for (;i!=seq.end(); ++i)
  {
    k = i->find(atom.c_str());

    if (k==string::npos)
      continue;

    current=i;
    return true;
  }

  return false;
}

tokenizer::operator stringc

(

)

Output the tokenizer back as a string.

Definition at line 480 of file tokenizer.cpp.

References reset().

{
  string s;
  tokenizer& tk(*this);
  for ( tk.reset(); !tk; ++tk)
    { s += tk(); }; 
  
  return s; 
}

boolc tokenizer::operator!

(

)

const

Can the stream be read?

Definition at line 8 of file tokenizer.cpp.

References current, and seq.

{ 
  return (current != seq.end()); 
}

stringc & tokenizer::operator()

(

)

const

Access the current token in the stream.

Definition at line 19 of file tokenizer.cpp.

References current, and seq.

{ 
  assert(current != seq.end()); 
  return *current; 
}

string & tokenizer::operator*

(

)

Modify the current token.

Definition at line 13 of file tokenizer.cpp.

References current, and seq.

{ 
  assert(current != seq.end()); 
  return *current; 
} 

void tokenizer::operator++

(

)

Increment the stream's index.

Definition at line 427 of file tokenizer.cpp.

References current, and seq.

{
  if (current==seq.end())
    return;

  ++current;
} 

boolc tokenizer::operator==

(

tokenizer &

t2

)

Data copy.

Compare tokenizers by comparing each token.

Definition at line 452 of file tokenizer.cpp.

References reset().

{
  reset();
  t2.reset();
  for ( ;!t2; ++t2 )
  {
    if (!(*this)==false)
      return false; 

    if ( (*this)() != t2() )
      return false;

    ++(*this);
  }

  if (!(*this))
    return false;

  return true;
}

ostream & tokenizer::print

(

ostream &

os

)

const

Print seq.

Set the printdelimiter to any string.

Definition at line 407 of file tokenizer.cpp.

References printdelimiter, and seq.

Referenced by operator<<().

{
  liststringic i = seq.begin();
  liststringic iend2 = seq.end();
  if (i!=iend2)
    os << *i;
  ++i;
  for ( ; i!=iend2; ++i )
  {
    os << printdelimiter << *i;
  }

  return os;
}

void tokenizer::read

(

stringc &

data

)

Read the data in.

Definition at line 46 of file tokenizer.cpp.

References seq.

Referenced by modulereport::reset().

{ 
  seq.push_back(data); 
}

void tokenizer::readaslines

(

stringc &

data

)

Read the string as lines.

Each line is a string.

Definition at line 51 of file tokenizer.cpp.

References seq, and subtract().

Referenced by modulelist::buildlist(), projunittest::eval(), html::insert(), readaslinesgeneral(), misclib_testcode::tokenizertest::test05(), misclib_testcode::tokenizertest::test06(), and misclib_testcode::tokenizertest::test08().

{ 
  seq.push_back(data); subtract("\n"); 
}

void tokenizer::readaslinesgeneral

(

stringc &

data

)

Because special characters in strings can interfere with reading strings, this routine is a hack to remove white spaces, commas and empty lines.

Definition at line 392 of file tokenizer.cpp.

References readaslines(), remove_if(), subtract(), and trim().

Referenced by projunittests::eval(), simplexD2tessindexed< PT, PD, INDX >::serializeInverse(), simplexD1tessindexed< PT, PD, INDX >::serializeInverse(), and simplexD1listlinked< VI, INDX >::serializeInverse().

{
  readaslines(data);
  subtract(",");
  subtract(" ");
  trim();
  remove_if();
}

void tokenizer::remove

(

stringc &

token

)

Remove matching tokens.

Definition at line 25 of file tokenizer.cpp.

References seq.

Referenced by misclib_testcode::tokenizertest::test04().

{ 
  seq.remove(token); 
}

template<typename SPACER >

void tokenizer::remove_if

(

SPACER

spacer

)

[inline]

Remove invalid lines.

Definition at line 112 of file tokenizer.h.

References seq.

    { seq.remove_if(spacer); }

void tokenizer::remove_if

(

)

Remove invalid lines as interpreted by myspacer.

Definition at line 30 of file tokenizer.cpp.

References seq.

Referenced by tokenizermisc::comparewithoutspace(), readaslinesgeneral(), misclib_testcode::tokenizertest::test06(), misclib_testcode::tokenizertest::test08(), misclib_testcode::tokenizertest::test09(), tokenize(), and trim_and_prune().

{ 
  seq.remove_if(spacerdelete<>()); 
}

void tokenizer::reset

(

)

Resets token iteration.

Definition at line 422 of file tokenizer.cpp.

References current, and seq.

Referenced by menusystem::addfont10paragraphs(), modulelist::buildlist(), modulereport::compile_exitstatus(), projunittests::eval(), projunittest::eval(), vispoint3::handlecommand(), vispoint2::handlecommand(), visline::handlecommand(), html::insert(), operator stringc(), operator==(), tokenizerlocal::reset(), simplexD2tessindexed< PT, PD, INDX >::serializeInverse(), simplexD1tessindexed< PT, PD, INDX >::serializeInverse(), simplexD1listlinked< VI, INDX >::serializeInverse(), gobjglColor3f::serializeInverse(), gobjglVertex3f::serializeInverse(), makestate< CFC >::standardbuild(), misclib_testcode::tokenizertest::test09(), misclib_testcode::tokenizertest::test10(), tokenizer(), misclib_testcode::tokenizertest::unittest01(), misclib_testcode::tokenizertest::unittest02(), misclib_testcode::tokenizertest::unittest03(), and modulereport::unittests_exitstatus().

{
  current=seq.begin();
}

void tokenizer::stripcomment

(

stringc &

comment

)

For each line delete right of and including the comment if it exists in the line.

Definition at line 123 of file tokenizer.cpp.

References seq.

Referenced by misclib_testcode::tokenizertest::test04(), misclib_testcode::tokenizertest::test06(), and misclib_testcode::tokenizertest::test08().

{
  liststringi k = seq.begin();
  string::size_type i;
  for (;k!=seq.end(); ++k)
  {
    string & token(*k);
    i=0;
    i = token.find(comment,i);
    if (i==string::npos)
      continue;

    token.erase(i);
  }
}

void tokenizer::subtract

(

stringc &

atom

)

Splits strings removing the atom.

Definition at line 401 of file tokenizer.cpp.

References seq.

Referenced by menusystem::addfont10blockstart(), menusystem::addfont10paragraphs(), vispoint3::handlecommand(), vispoint2::handlecommand(), visline::handlecommand(), readaslines(), readaslinesgeneral(), makestate< CFC >::standardbuild(), misclib_testcode::tokenizertest::test00(), misclib_testcode::tokenizertest::test04(), misclib_testcode::tokenizertest::test06(), misclib_testcode::tokenizertest::test08(), and tokenize().

{
  atomize(atom);
  seq.remove(atom);
}

void tokenizer::tokenize

(

)

delimiter set.

Definition at line 473 of file tokenizer.cpp.

References remove_if(), subtract(), and trim().

Referenced by gobjglColor3f::serializeInverse(), and gobjglVertex3f::serializeInverse().

{
  subtract(" ");
  trim();
  remove_if();
}

void tokenizer::trim

(

)

Trim the leading and trailing space of each token.

Definition at line 35 of file tokenizer.cpp.

References apply().

Referenced by tokenizermisc::comparewithoutspace(), readaslinesgeneral(), misclib_testcode::tokenizertest::test07(), misclib_testcode::tokenizertest::test08(), misclib_testcode::tokenizertest::test09(), tokenize(), and trim_and_prune().

{ 
  apply(spacertrim<>()); 
}

void tokenizer::trim_and_prune

(

)

Remove surrounding white space and delete empty strings.

Definition at line 40 of file tokenizer.cpp.

References remove_if(), and trim().

Referenced by projunittest::eval(), misclib_testcode::tokenizertest::test09(), and misclib_testcode::tokenizertest::unittest01().

{ 
  trim(); 
  remove_if(spacerdelete<>()); 
}

---

Member Data Documentation

liststringi tokenizer::current

The current state.

Definition at line 73 of file tokenizer.h.

Referenced by tokenizerlocal::erasetag(), html::insert(), tokenizerlocal::operator!(), operator!(), tokenizerlocal::operator()(), operator()(), operator*(), tokenizerlocal::operator++(), operator++(), tokenizerlocal::reset(), reset(), tokenizerlocal::scopesearch(), tokenizer(), tokenizerlocal::tokenizerlocal(), misclib_testcode::tokenizertest::unittest01(), and tokenizerlocal::writetag().

string tokenizer::printdelimiter

Each element printed is separated by the delimiter.

Definition at line 147 of file tokenizer.h.

Referenced by print(), misclib_testcode::tokenizertest::test04(), misclib_testcode::tokenizertest::test06(), misclib_testcode::tokenizertest::test07(), and misclib_testcode::tokenizertest::test08().

list<string> tokenizer::seq

Data representation.

Definition at line 76 of file tokenizer.h.

Referenced by menusystem::addfont10blockstart(), apply(), atomize(), tokenizerlocal::endpointreset(), tokenizerlocal::erasetag(), vispoint3::handlecommand(), vispoint2::handlecommand(), visline::handlecommand(), html::insert(), operator!(), operator()(), operator*(), operator++(), print(), read(), readaslines(), remove(), remove_if(), reset(), modulereport::reset(), gobjglColor3f::serializeInverse(), gobjglVertex3f::serializeInverse(), stripcomment(), subtract(), misclib_testcode::tokenizertest::test00(), misclib_testcode::tokenizertest::test03(), misclib_testcode::tokenizertest::test04(), misclib_testcode::tokenizertest::test07(), misclib_testcode::tokenizertest::test10(), tokenizer(), tokenizerlocal::tokenizerlocal(), misclib_testcode::tokenizertest::unittest01(), misclib_testcode::tokenizertest::unittest02(), and tokenizerlocal::writetag().

---

The documentation for this class was generated from the following files:

• misclib/tokenizer.h
• misclib/tokenizer.cpp