CMUCL commit: src (15 files)

[Raymond Toy]

    Date: Wednesday, September 15, 2010 @ 17:06:39
  Author: rtoy
    Path: /project/cmucl/cvsroot/src

Modified: code/string.lisp code/unidata.lisp i18n/BidiMirroring.txt
          i18n/CaseFolding.txt i18n/CompositionExclusions.txt
          i18n/DerivedNormalizationProps.txt i18n/NameAliases.txt
          i18n/NormalizationCorrections.txt i18n/SpecialCasing.txt
          i18n/UnicodeData.txt i18n/WordBreakProperty.txt
          i18n/tests/NormalizationTest.txt i18n/tests/WordBreakTest.txt
          i18n/unidata.bin tools/build-unidata.lisp

Add support for Unicode 5.2.  The normalization and wordbreak tests pass.

code/string.lisp:
o In %compose, handle the case where the composite character is
  outside the BMP and thus needs special handling for our UTF-16
  strings.

code/unidata.lisp
o CKJ Ideograph range has changed in 5.2.
o Fix bug in build-composition-table.  We were not correctly handling
  the case where the decomposition of a codepoint was outside the
  BMP.  Special care is needed to handle the UTF-16 strings that we
  use.
o The key for the pairwise composition table are the full codepoints,
  so we need to shift one by 21 bits instead of 16.

tools/build-unidata.lisp
o Update minor version to 2.

i18n/BidiMirroring.txt
i18n/CaseFolding.txt
i18n/CompositionExclusions.txt
i18n/DerivedNormalizationProps.txt
i18n/NameAliases.txt
i18n/NormalizationCorrections.txt
i18n/SpecialCasing.txt
i18n/UnicodeData.txt
i18n/WordBreakProperty.txt
i18n/tests/NormalizationTest.txt
i18n/tests/WordBreakTest.txt
o Updated from Unicode 5.2.

i18n/unidata.bin
o Regenerated from new Unicode 5.2 files.


------------------------------------+
 code/string.lisp                   |   20 
 code/unidata.lisp                  |   52 
 i18n/BidiMirroring.txt             |   10 
 i18n/CaseFolding.txt               |   14 
 i18n/CompositionExclusions.txt     |   10 
 i18n/DerivedNormalizationProps.txt | 6439 ++++++++++++++++++++++++++++++++++-
 i18n/NameAliases.txt               |    8 
 i18n/NormalizationCorrections.txt  |    6 
 i18n/SpecialCasing.txt             |   18 
 i18n/UnicodeData.txt               | 3377 +++++++++++++++---
 i18n/WordBreakProperty.txt         |  132 
 i18n/tests/NormalizationTest.txt   |  305 +
 i18n/tests/WordBreakTest.txt       |   10 
 i18n/unidata.bin                   |    <<Binary file>>
 tools/build-unidata.lisp           |    4 
 15 files changed, 9842 insertions(+), 564 deletions(-)


The diff is longer than the limit of 200KB.