In-depth The Scanner

| In-depth: The Scanner |

## Basic Scanner API

class scanner
`value_t` | typedef: The value type of the scanner's iterator
| `ref_t` | typedef: The reference type of the scanner's iterator | `bool at_end() const` | Returns true if the input is exhausted | `value_t operator*() const` | Dereference/get a `value_t` from the input | `scanner const& operator++()` | move the scanner forward
`IteratorT& first` | The iterator pointing to the current input position. Held by reference
`IteratorT const last` | The iterator pointing to the end of the input. Held by value

The basic behavior of the scanner is handled by policies. The actual execution of the scanner's public member functions listed in the table above is implemented by the scanner policies.

Three sets of policies govern the behavior of the scanner. These policies make it possible to extend Spirit non-intrusively. The scanner policies allow the core-functionality to be extended without requiring any potentially destabilizing changes to the code. A library writer might provide her own policies that override the ones that are already in place to fine tune the parsing process to fit her own needs. Layers above the core might also want to take advantage of this policy based machanism. Abstract syntax tree generation, debuggers and lexers come to mind.

There are three sets of policies that govern:

- Iteration and filtering
- Recognition and matching
- Handling semantic actions

## iteration_policy

Here are the default policies that govern iteration and filtering:

        struct iteration_policy
        {
            template <typename ScannerT>
            void
            advance(ScannerT const& scan) const
            { ++scan.first; }

            template <typename ScannerT>
            bool at_end(ScannerT const& scan) const
            { return scan.first == scan.last; }

            template <typename T>
            T filter(T ch) const
            { return ch; }

            template <typename ScannerT>
            typename ScannerT::ref_t
            get(ScannerT const& scan) const
            { return *scan.first; }
        };

Iteration and filtering policies
advance | Move the iterator forward
| at_end | Return true if the input is exhausted | filter | Filter a character read from the input | get | Read a character from the input

The following code snippet demonstrates a simple policy that converts all characters to lower case:

        struct inhibit_case_iteration_policy : public iteration_policy
        {
            template <typename CharT>
            CharT filter(CharT ch) const
            {
                return std::tolower(ch);
            }
        };

## match_policy

Here are the default policies that govern recognition and matching:

        struct match_policy
        {
            template <typename T>
            struct result
            {
                typedef match<T> type;
            };

            const match<nil_t>
            no_match() const
            {
                return match<nil_t>();
            }

            const match<nil_t>
            empty_match() const
            {
                return match<nil_t>(0, nil_t());
            }

            template <typename AttrT, typename IteratorT>
            match<AttrT>
            create_match(
                std::size_t         length,
                AttrT const&        val,
                IteratorT const&    /*first*/,
                IteratorT const&    /*last*/) const
            {
                return match<AttrT>(length, val);
            }

            template <typename MatchT, typename IteratorT>
            void
            group_match(
                MatchT&             /*m*/,
                parser_id const&    /*id*/,
                IteratorT const&    /*first*/,
                IteratorT const&    /*last*/) const {}

            template <typename Match1T, typename Match2T>
            void
            concat_match(Match1T& l, Match2T const& r) const
            {
                l.concat(r);
            }
        };

Recognition and matching
result | A metafunction that returns a match type given an attribute type (see In-depth: The Parser)
| no_match | Create a failed match | empty_match | Create an empty match. An empty match is a successful epsilon match (matching length == 0) | create_match | Create a match given the matching length, an attribute and the iterator pair pointing to the matching portion of the input | group_match | For non terminals such as rules, this is called after a successful match has been made to allow post processing | concat_match | Concatenate two match objects

## action_policy

The action policy has only one function for handling semantic actions:

        struct action_policy
        {
            template <typename ActorT, typename AttrT, typename IteratorT>
            void
            do_action(
                ActorT const&       actor,
                AttrT const&        val,
                IteratorT const&    first,
                IteratorT const&    last) const;
        };

The default action policy forwards to:

        actor(first, last);

If the attribute `val` is of type nil_t. Otherwise:

        actor(val);

### scanner_policies mixer

The class `scanner_policies` combines the three scanner policy classes above into one:

        template <
            typename IterationPolicyT   = iteration_policy,
            typename MatchPolicyT       = match_policy,
            typename ActionPolicyT      = action_policy>
        struct scanner_policies;

This mixer class inherits from all the three policies. This scanner_policies class is then used to parameterize the scanner:

        template <
            typename IteratorT = char const*,
            typename PoliciesT = scanner_policies<> >
        class scanner;

The scanner in turn inherits from the PoliciesT.

### Rebinding Policies

The scanner can be made to rebind to a different set of policies anytime. It has a member function `change_policies(new_policies)`. Given a new set of policies, this member function creates a new scanner with the new set of policies. The result type of the rebound scanner can be can be obtained by calling the metafunction:

        rebind_scanner_policies<ScannerT, PoliciesT>::type

### Rebinding Iterators

The scanner can also be made to rebind to a different iterator type anytime. It has a member function `change_iterator(first, last)`. Given a new pair of iterator of type different from the ones held by the scanner, this member function creates a new scanner with the new pair of iterators. The result type of the rebound scanner can be can be obtained by calling the metafunction:

        rebind_scanner_iterator<ScannerT, IteratorT>::type

Copyright © 1998-2003 Joel de Guzman
Use, modification and distribution is subject to the Boost Software License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)