tools: Check in tools for shrinking ICU size, change default to small-icu

* Change configure default to "small-icu" (Intl on, English only)
* add "--without-intl" and "vcbuild without-intl" options, equivalent
to --with-intl=none
* update BUILDING.md with above changes
* Checks in tools that generate the deps/icu-small source directory
from ICU source
* Tools and process for updating ICU documented in tools/icu/README.md

Fixes: https://github.com/nodejs/node/issues/3476
PR-URL: https://github.com/nodejs/node/pull/6088
Reviewed-By: James M Snell <jasnell@gmail.com>

3 files changed, 326 insertions, 72 deletions

diff --git a/tools/icu/README.md b/tools/icu/README.md
index 40d5287d9f..6c08bf1598 100644
--- a/tools/icu/README.md
+++ b/tools/icu/README.md
@@ -1,6 +1,84 @@
 Notes about the icu directory.
 ===
 
+How to upgrade ICU
+---
+
+- Make sure your node workspace is clean (clean `git status`) should be sufficient.
+- Configure Node with the specific [ICU version](http://icu-project.org/download) you want to upgrade to, for example:
+
+```
+./configure \
+    --with-intl=small-icu \
+    --with-icu-source=http://download.icu-project.org/files/icu4c/56.1/icu4c-56_1-src.zip
+make
+```
+
+(the equivalent `vcbuild.bat` commands should work also.)
+
+- (note- may need to make changes in `icu-generic.gyp` or `tools/icu/patches` for
+version specific stuff)
+
+- Verify the node build works
+
+```
+make test-ci
+```
+
+Also running
+```js
+ new Intl.DateTimeFormat('es',{month:'long'}).format(new Date(9E8));
+```
+
+…Should return `January` not `enero`.
+(TODO here: improve [testing](https://github.com/nodejs/Intl/issues/16))
+
+
+- Now, copy `deps/icu` over to `deps/icu-small`
+
+```
+python tools/icu/shrink-icu-src.py
+```
+
+- Now, do a clean rebuild of node to test:
+
+(TODO: fix this when these options become default)
+
+```
+./configure --with-intl=small-icu --with-icu-source=deps/icu-small
+make
+```
+
+- Test this newly default-generated Node.js
+```js
+process.versions.icu;
+new Intl.DateTimeFormat('es',{month:'long'}).format(new Date(9E8));
+```
+
+(should return your updated ICU version number, and also `January` again.)
+
+- You are ready to check in the updated `deps/small-icu`.
+This is a big commit, so make this a separate commit from other changes.
+
+- Now, fix the default URL for the `full-icu` build in `/configure`, in
+the `configure_intl()` function. It should match the ICU URL used in the
+first step.  When this is done, the following should build with full ICU.
+
+```
+# clean up
+rm -rf out deps/icu deps/icu4c*
+./configure --with-intl=full-icu --download=all
+make
+make test-ci
+```
+
+- commit the change to `configure`.
+
+-----
+
+Notes about these tools
+---
+
 The files in this directory were written for the node.js effort. It's
 the intent of their author (Steven R. Loomis / srl295) to merge them
 upstream into ICU, pending much discussion within the ICU-PMC.
diff --git a/tools/icu/icu-generic.gyp b/tools/icu/icu-generic.gyp
index 222a9e9566..a61b294141 100644
--- a/tools/icu/icu-generic.gyp
+++ b/tools/icu/icu-generic.gyp
@@ -8,8 +8,8 @@
 {
   'variables': {
     'icu_src_derb': [
-      '../../deps/icu/source/tools/genrb/derb.c',
-      '../../deps/icu/source/tools/genrb/derb.cpp'
+      '<(icu_path)/source/tools/genrb/derb.c',
+      '<(icu_path)/source/tools/genrb/derb.cpp'
     ],
   },
   'includes': [ '../../icu_config.gypi' ],
@@ -121,49 +121,82 @@
           'sources': [
             '<@(icu_src_i18n)'
           ],
+          ## if your compiler can dead-strip, these exclusions will
+          ## make ZERO difference to binary size.
+          ## Made ICU-specific for future-proofing.
           'conditions': [
             [ 'icu_ver_major == 55', { 'sources!': [
-              ## Strip out the following for ICU 55 only.
-              ## add more conditions in the future?
-              ## if your compiler can dead-strip, this will
-              ## make ZERO difference to binary size.
-              ## Made ICU-specific for future-proofing.
+              # alphabetic index
+              '<(icu_path)/source/i18n/alphaindex.cpp',
+              # BOCSU
+              # misc
+              '<(icu_path)/source/i18n/regexcmp.cpp',
+              '<(icu_path)/source/i18n/regexcmp.h',
+              '<(icu_path)/source/i18n/regexcst.h',
+              '<(icu_path)/source/i18n/regeximp.cpp',
+              '<(icu_path)/source/i18n/regeximp.h',
+              '<(icu_path)/source/i18n/regexst.cpp',
+              '<(icu_path)/source/i18n/regexst.h',
+              '<(icu_path)/source/i18n/regextxt.cpp',
+              '<(icu_path)/source/i18n/regextxt.h',
+              '<(icu_path)/source/i18n/region.cpp',
+              '<(icu_path)/source/i18n/region_impl.h',
+              '<(icu_path)/source/i18n/reldatefmt.cpp',
+              '<(icu_path)/source/i18n/reldatefmt.h'
+              '<(icu_path)/source/i18n/scientificformathelper.cpp',
+              '<(icu_path)/source/i18n/tmunit.cpp',
+              '<(icu_path)/source/i18n/tmutamt.cpp',
+              '<(icu_path)/source/i18n/tmutfmt.cpp',
+              '<(icu_path)/source/i18n/uregex.cpp',
+              '<(icu_path)/source/i18n/uregexc.cpp',
+              '<(icu_path)/source/i18n/uregion.cpp',
+              '<(icu_path)/source/i18n/uspoof.cpp',
+              '<(icu_path)/source/i18n/uspoof_build.cpp',
+              '<(icu_path)/source/i18n/uspoof_conf.cpp',
+              '<(icu_path)/source/i18n/uspoof_conf.h',
+              '<(icu_path)/source/i18n/uspoof_impl.cpp',
+              '<(icu_path)/source/i18n/uspoof_impl.h',
+              '<(icu_path)/source/i18n/uspoof_wsconf.cpp',
+              '<(icu_path)/source/i18n/uspoof_wsconf.h',
+            ]}],
+            [ 'icu_ver_major == 57', { 'sources!': [
 
               # alphabetic index
-              '../../deps/icu/source/i18n/alphaindex.cpp',
+              '<(icu_path)/source/i18n/alphaindex.cpp',
               # BOCSU
               # misc
-              '../../deps/icu/source/i18n/regexcmp.cpp',
-              '../../deps/icu/source/i18n/regexcmp.h',
-              '../../deps/icu/source/i18n/regexcst.h',
-              '../../deps/icu/source/i18n/regeximp.cpp',
-              '../../deps/icu/source/i18n/regeximp.h',
-              '../../deps/icu/source/i18n/regexst.cpp',
-              '../../deps/icu/source/i18n/regexst.h',
-              '../../deps/icu/source/i18n/regextxt.cpp',
-              '../../deps/icu/source/i18n/regextxt.h',
-              '../../deps/icu/source/i18n/region.cpp',
-              '../../deps/icu/source/i18n/region_impl.h',
-              '../../deps/icu/source/i18n/reldatefmt.cpp',
-              '../../deps/icu/source/i18n/reldatefmt.h'
-              '../../deps/icu/source/i18n/scientificformathelper.cpp',
-              '../../deps/icu/source/i18n/tmunit.cpp',
-              '../../deps/icu/source/i18n/tmutamt.cpp',
-              '../../deps/icu/source/i18n/tmutfmt.cpp',
-              '../../deps/icu/source/i18n/uregex.cpp',
-              '../../deps/icu/source/i18n/uregexc.cpp',
-              '../../deps/icu/source/i18n/uregion.cpp',
-              '../../deps/icu/source/i18n/uspoof.cpp',
-              '../../deps/icu/source/i18n/uspoof_build.cpp',
-              '../../deps/icu/source/i18n/uspoof_conf.cpp',
-              '../../deps/icu/source/i18n/uspoof_conf.h',
-              '../../deps/icu/source/i18n/uspoof_impl.cpp',
-              '../../deps/icu/source/i18n/uspoof_impl.h',
-              '../../deps/icu/source/i18n/uspoof_wsconf.cpp',
-              '../../deps/icu/source/i18n/uspoof_wsconf.h',
-            ]}]],
+              '<(icu_path)/source/i18n/regexcmp.cpp',
+              '<(icu_path)/source/i18n/regexcmp.h',
+              '<(icu_path)/source/i18n/regexcst.h',
+              '<(icu_path)/source/i18n/regeximp.cpp',
+              '<(icu_path)/source/i18n/regeximp.h',
+              '<(icu_path)/source/i18n/regexst.cpp',
+              '<(icu_path)/source/i18n/regexst.h',
+              '<(icu_path)/source/i18n/regextxt.cpp',
+              '<(icu_path)/source/i18n/regextxt.h',
+              '<(icu_path)/source/i18n/region.cpp',
+              '<(icu_path)/source/i18n/region_impl.h',
+              '<(icu_path)/source/i18n/reldatefmt.cpp',
+              '<(icu_path)/source/i18n/reldatefmt.h'
+              '<(icu_path)/source/i18n/scientificformathelper.cpp',
+              '<(icu_path)/source/i18n/tmunit.cpp',
+              '<(icu_path)/source/i18n/tmutamt.cpp',
+              '<(icu_path)/source/i18n/tmutfmt.cpp',
+              '<(icu_path)/source/i18n/uregex.cpp',
+              '<(icu_path)/source/i18n/uregexc.cpp',
+              '<(icu_path)/source/i18n/uregion.cpp',
+              '<(icu_path)/source/i18n/uspoof.cpp',
+              '<(icu_path)/source/i18n/uspoof_build.cpp',
+              '<(icu_path)/source/i18n/uspoof_conf.cpp',
+              '<(icu_path)/source/i18n/uspoof_conf.h',
+              '<(icu_path)/source/i18n/uspoof_impl.cpp',
+              '<(icu_path)/source/i18n/uspoof_impl.h',
+              '<(icu_path)/source/i18n/uspoof_wsconf.cpp',
+              '<(icu_path)/source/i18n/uspoof_wsconf.h',
+            ]}],
+            ],
           'include_dirs': [
-            '../../deps/icu/source/i18n',
+            '<(icu_path)/source/i18n',
           ],
           'defines': [
             'U_I18N_IMPLEMENTATION=1',
@@ -171,7 +204,7 @@
           'dependencies': [ 'icuucx', 'icu_implementation', 'icu_uconfig', 'icu_uconfig_target' ],
           'direct_dependent_settings': {
             'include_dirs': [
-              '../../deps/icu/source/i18n',
+              '<(icu_path)/source/i18n',
             ],
           },
           'export_dependent_settings': [ 'icuucx', 'icu_uconfig_target' ],
@@ -253,7 +286,7 @@
               'sources': [ '<(SHARED_INTERMEDIATE_DIR)/icudt<(icu_ver_major)_dat.c' ],
               'dependencies': [ 'genccode#host', 'icupkg#host', 'icu_implementation#host', 'icu_uconfig' ],
               'include_dirs': [
-                '../../deps/icu/source/common',
+                '<(icu_path)/source/common',
               ],
               'actions': [
                 {
@@ -333,7 +366,7 @@
               'sources': [ '<(SHARED_INTERMEDIATE_DIR)/icusmdt<(icu_ver_major)_dat.c' ],
               # for umachine.h
               'include_dirs': [
-                '../../deps/icu/source/common',
+                '<(icu_path)/source/common',
               ],
             }]], # end icu_small == true
         }]], # end OS != win
@@ -349,7 +382,7 @@
         '<@(icu_src_stubdata)'
       ],
       'include_dirs': [
-        '../../deps/icu/source/common',
+        '<(icu_path)/source/common',
       ],
     },
     # this target is for v8 consumption.
@@ -379,34 +412,51 @@
       'sources': [
         '<@(icu_src_common)',
       ],
-      'conditions': [
-        [ 'icu_ver_major == 55', { 'sources!': [
-          ## Strip out the following for ICU 55 only.
-          ## add more conditions in the future?
           ## if your compiler can dead-strip, this will
           ## make ZERO difference to binary size.
           ## Made ICU-specific for future-proofing.
+      'conditions': [
+        [ 'icu_ver_major == 55', { 'sources!': [
+
+          # bidi- not needed (yet!)
+          '<(icu_path)/source/common/ubidi.c',
+          '<(icu_path)/source/common/ubidiimp.h',
+          '<(icu_path)/source/common/ubidiln.c',
+          '<(icu_path)/source/common/ubidiwrt.c',
+          #'<(icu_path)/source/common/ubidi_props.c',
+          #'<(icu_path)/source/common/ubidi_props.h',
+          #'<(icu_path)/source/common/ubidi_props_data.h',
+          # and the callers
+          '<(icu_path)/source/common/ushape.cpp',
+          '<(icu_path)/source/common/usprep.cpp',
+          '<(icu_path)/source/common/uts46.cpp',
+          '<(icu_path)/source/common/uidna.cpp',
+        ]}],
+        [ 'icu_ver_major == 57', { 'sources!': [
+          # work around http://bugs.icu-project.org/trac/ticket/12451
+          # (benign afterwards)
+          '<(icu_path)/source/common/cstr.cpp',
 
           # bidi- not needed (yet!)
-          '../../deps/icu/source/common/ubidi.c',
-          '../../deps/icu/source/common/ubidiimp.h',
-          '../../deps/icu/source/common/ubidiln.c',
-          '../../deps/icu/source/common/ubidiwrt.c',
-          #'../../deps/icu/source/common/ubidi_props.c',
-          #'../../deps/icu/source/common/ubidi_props.h',
-          #'../../deps/icu/source/common/ubidi_props_data.h',
+          '<(icu_path)/source/common/ubidi.c',
+          '<(icu_path)/source/common/ubidiimp.h',
+          '<(icu_path)/source/common/ubidiln.c',
+          '<(icu_path)/source/common/ubidiwrt.c',
+          #'<(icu_path)/source/common/ubidi_props.c',
+          #'<(icu_path)/source/common/ubidi_props.h',
+          #'<(icu_path)/source/common/ubidi_props_data.h',
           # and the callers
-          '../../deps/icu/source/common/ushape.cpp',
-          '../../deps/icu/source/common/usprep.cpp',
-          '../../deps/icu/source/common/uts46.cpp',
-          '../../deps/icu/source/common/uidna.cpp',
+          '<(icu_path)/source/common/ushape.cpp',
+          '<(icu_path)/source/common/usprep.cpp',
+          '<(icu_path)/source/common/uts46.cpp',
+          '<(icu_path)/source/common/uidna.cpp',
         ]}],
         [ 'OS == "solaris"', { 'defines': [
           '_XOPEN_SOURCE_EXTENDED=0',
         ]}],
       ],
       'include_dirs': [
-        '../../deps/icu/source/common',
+        '<(icu_path)/source/common',
       ],
       'defines': [
         'U_COMMON_IMPLEMENTATION=1',
@@ -415,7 +465,7 @@
       'export_dependent_settings': [ 'icu_uconfig', 'icu_uconfig_target' ],
       'direct_dependent_settings': {
         'include_dirs': [
-          '../../deps/icu/source/common',
+          '<(icu_path)/source/common',
         ],
         'conditions': [
           [ 'OS=="win"', {
@@ -426,7 +476,7 @@
         ],
       },
     },
-    # tools library
+    # tools library. This builds all of ICU together.
     {
       'target_name': 'icutools',
       'type': '<(library)',
@@ -440,16 +490,16 @@
         '<@(icu_src_stubdata)',
       ],
       'sources!': [
-        '../../deps/icu/source/tools/toolutil/udbgutil.cpp',
-        '../../deps/icu/source/tools/toolutil/udbgutil.h',
-        '../../deps/icu/source/tools/toolutil/dbgutil.cpp',
-        '../../deps/icu/source/tools/toolutil/dbgutil.h',
+        '<(icu_path)/source/tools/toolutil/udbgutil.cpp',
+        '<(icu_path)/source/tools/toolutil/udbgutil.h',
+        '<(icu_path)/source/tools/toolutil/dbgutil.cpp',
+        '<(icu_path)/source/tools/toolutil/dbgutil.h',
       ],
       'include_dirs': [
-        '../../deps/icu/source/common',
-        '../../deps/icu/source/i18n',
-        '../../deps/icu/source/io',
-        '../../deps/icu/source/tools/toolutil',
+        '<(icu_path)/source/common',
+        '<(icu_path)/source/i18n',
+        '<(icu_path)/source/io',
+        '<(icu_path)/source/tools/toolutil',
       ],
       'defines': [
         'U_COMMON_IMPLEMENTATION=1',
@@ -466,10 +516,10 @@
       ],
       'direct_dependent_settings': {
         'include_dirs': [
-          '../../deps/icu/source/common',
-          '../../deps/icu/source/i18n',
-          '../../deps/icu/source/io',
-          '../../deps/icu/source/tools/toolutil',
+          '<(icu_path)/source/common',
+          '<(icu_path)/source/i18n',
+          '<(icu_path)/source/io',
+          '<(icu_path)/source/tools/toolutil',
         ],
         'conditions': [
           [ 'OS=="win"', {
diff --git a/tools/icu/shrink-icu-src.py b/tools/icu/shrink-icu-src.py
new file mode 100644
index 0000000000..1760b4ff47
--- /dev/null
+++ b/tools/icu/shrink-icu-src.py
@@ -0,0 +1,126 @@
+#!/usr/bin/env python
+import optparse
+import os
+import pprint
+import re
+import shlex
+import subprocess
+import sys
+import shutil
+import string
+
+parser = optparse.OptionParser()
+
+parser.add_option('--icu-small',
+    action='store',
+    dest='icusmall',
+    default='deps/icu-small',
+    help='path to target ICU directory to shrink. Will be deleted.')
+
+parser.add_option('--icu-src',
+    action='store',
+    dest='icusrc',
+    default='deps/icu',
+    help='path to source ICU directory.')
+
+parser.add_option('--icutmp',
+    action='store',
+    dest='icutmp',
+    default='out/Release/gen/icutmp',
+    help='path to icutmp dir.')
+
+
+(options, args) = parser.parse_args()
+
+if os.path.isdir(options.icusmall):
+    print 'Deleting existing icusmall %s' % (options.icusmall)
+    shutil.rmtree(options.icusmall)
+
+if not os.path.isdir(options.icusrc):
+    print 'Missing source ICU dir --icusrc=%' % (options.icusrc)
+    sys.exit(1)
+
+
+
+ignore_regex = re.compile('^.*\.(vcxproj|filters|nrm|icu|dat|xml|txt|ac|guess|m4|in|sub|py|mak)$')
+
+def icu_ignore(dir, files):
+    subdir = dir[len(options.icusrc)+1::]
+    ign = []
+    if len(subdir) == 0:
+        # remove all files at root level
+        ign = ign + files
+        # except...
+        ign.remove('source')
+        ign.remove('license.html')
+        ign.remove('LICENSE')
+    elif subdir == 'source':
+        ign = ign + ['layout','samples','test','extra','config','layoutex','allinone']
+        ign = ign + ['runConfigureICU','install-sh','mkinstalldirs','configure']
+    elif subdir == 'source/tools':
+        ign = ign + ['tzcode','ctestfw','gensprep','gennorm2','gendict','icuswap',
+        'genbrk','gencfu','gencolusb','genren','memcheck','makeconv','gencnval','icuinfo','gentest']
+    elif subdir == 'source/data':
+        ign = ign + ['unidata','curr','zone','unit','lang','region','misc','sprep']
+    # else:
+        # print '!%s! [%s]' % (subdir, files)
+    ign = ign + ['.DS_Store', 'Makefile', 'Makefile.in']
+
+    for file in files:
+        if ignore_regex.match(file):
+            ign = ign + [file]
+
+    # print '>%s< [%s]' % (subdir, ign)
+    return ign
+
+# copied from configure
+def icu_info(icu_full_path):
+    uvernum_h = os.path.join(icu_full_path, 'source/common/unicode/uvernum.h')
+    if not os.path.isfile(uvernum_h):
+        print ' Error: could not load %s - is ICU installed?' % uvernum_h
+        sys.exit(1)
+    icu_ver_major = None
+    matchVerExp = r'^\s*#define\s+U_ICU_VERSION_SHORT\s+"([^"]*)".*'
+    match_version = re.compile(matchVerExp)
+    for line in open(uvernum_h).readlines():
+        m = match_version.match(line)
+        if m:
+            icu_ver_major = m.group(1)
+    if not icu_ver_major:
+        print ' Could not read U_ICU_VERSION_SHORT version from %s' % uvernum_h
+        sys.exit(1)
+    icu_endianness = sys.byteorder[0];  # TODO(srl295): EBCDIC should be 'e'
+    return (icu_ver_major, icu_endianness)
+
+(icu_ver_major, icu_endianness) = icu_info(options.icusrc)
+print "icudt%s%s" % (icu_ver_major, icu_endianness)
+
+src_datafile = os.path.join(options.icutmp, "icusmdt%s.dat" % (icu_ver_major))
+dst_datafile = os.path.join(options.icusmall, "source","data","in", "icudt%s%s.dat" % (icu_ver_major, icu_endianness))
+
+if not os.path.isfile(src_datafile):
+    print "Could not find source datafile %s - did you build small-icu node?" % src_datafile
+    sys.exit(1)
+else:
+    print "will use small datafile %s" % (src_datafile)
+print '%s --> %s' % (options.icusrc, options.icusmall)
+shutil.copytree(options.icusrc, options.icusmall, ignore=icu_ignore)
+print '%s --> %s' % (src_datafile, dst_datafile)
+
+# OK, now copy the data file
+shutil.copy(src_datafile, dst_datafile)
+
+# Now, print a short notice
+readme_name = os.path.join(options.icusmall, "README-SMALL-ICU.txt" )
+
+fi = open(readme_name, 'wb')
+print >>fi, "Small ICU sources - auto generated by shrink-icu-src.py"
+print >>fi, ""
+print >>fi, "This directory contains the ICU subset used by --with-intl=small-icu (the default)"
+print >>fi, "It is a strict subset of ICU %s source files with the following exception(s):" % (icu_ver_major)
+print >>fi, "* %s : Reduced-size data file" % (dst_datafile)
+print >>fi, ""
+print >>fi, ""
+print >>fi, "To rebuild this directory, see ../../tools/icu/README.md"
+print >>fi, ""
+fi.close()