Andreas Ley a1718ae350 Fix for incorrect tokenization due to index difference of Unicode character/scalar (#286) ...

* Fix: `Scanner` now uses indices of the respective UnicodeScalarView

* Fix: `Scanner` now uses indices of the respective UnicodeScalarView

* Extended test for Unicode `Combining Diaeresis`

* Fixed test for combining diaeresis

* Inlined template for testing Unicode combining diaeresis

Co-authored-by: Ilya Puchka <ilyapuchka@gmail.com>

2020-01-12 15:21:02 +00:00

8.1 KiB

Raw Blame History

View Raw