build: i18n: add icu config options

Make "--with-intl=none" the default and add "intl-none" option to
vcbuild.bat.

If icu data is missing print a warning unless either --download=all or
--download=icu is set. If set then automatically download, verify (MD5)
and unpack the ICU data if not already available.

There's a "list" of URLs being used, but right now only the first is
picked up. The logic works something like this:

* If there is no directory deps/icu,
  * If no zip file (currently icu4c-54_1-src.zip),
    * Download zip file (icu-project.org -> sf.net)
  * Verify the MD5 sum of the zipfile
    * If bad, print error and exit
  * Unpack the zipfile into deps/icu
* If deps/icu now exists, use it, else fail with help text

Add the configuration option "--with-icu-source=..."

Usage:
  * --with-icu-source=/path/to/my/other/icu
  * --with-icu-source=/path/to/icu54.zip
  * --with-icu-source=/path/to/icu54.tgz
  * --with-icu-source=http://example.com/icu54.tar.bz2

Add the configuration option "--with-icu-locals=...".  Allows choosing
which locales are used in the "small-icu" case.

Example:
    configure --with-intl=small-icu --with-icu-locales=tlh,grc,nl

(Also note that as of this writing, neither Klingon nor Ancient Greek
are in upstream CLDR data. Serving suggestion only.)

Don't use hard coded ../../out paths on windows. This was suggested by
@misterdjules as it causes test failures.  With this fix, "out" is no
longer created on windows and the following can run properly:

    python tools/test.py simple

Reduce space by about 1MB with ICU 54 (over without this patch). Also
trims a few other source files, but only conditional on the exact ICU
version used. This is to future-proof - a file that is unneeded now may
be needed in future ICUs.

Also:
  * Update distclean to remove icu related files
  * Refactor some code into tools/configure.d/nodedownload.py
  * Update docs
  * Add test

PR-URL: https://github.com/joyent/node/pull/8719
Fixes: https://github.com/joyent/node/issues/7676#issuecomment-64704230
[trev.norris@gmail.com small change to test's whitespace and logic]
Signed-off-by: Trevor Norris <trev.norris@gmail.com>

1 files changed, 127 insertions, 0 deletions

diff --git a/tools/configure.d/nodedownload.py b/tools/configure.d/nodedownload.py
new file mode 100644
index 0000000000..e24efd865f
--- /dev/null
+++ b/tools/configure.d/nodedownload.py
@@ -0,0 +1,127 @@
+#!/usr/bin/env python
+# Moved some utilities here from ../../configure
+
+import urllib
+import hashlib
+import sys
+import zipfile
+import tarfile
+import fpformat
+import contextlib
+
+def formatSize(amt):
+    """Format a size as a string in MB"""
+    return fpformat.fix(amt / 1024000., 1)
+
+def spin(c):
+    """print out an ASCII 'spinner' based on the value of counter 'c'"""
+    spin = ".:|'"
+    return (spin[c % len(spin)])
+
+class ConfigOpener(urllib.FancyURLopener):
+    """fancy opener used by retrievefile. Set a UA"""
+    # append to existing version (UA)
+    version = '%s node.js/configure' % urllib.URLopener.version
+
+def reporthook(count, size, total):
+    """internal hook used by retrievefile"""
+    sys.stdout.write(' Fetch: %c %sMB total, %sMB downloaded   \r' %
+                     (spin(count),
+                      formatSize(total),
+                      formatSize(count*size)))
+
+def retrievefile(url, targetfile):
+    """fetch file 'url' as 'targetfile'. Return targetfile or throw."""
+    try:
+        sys.stdout.write(' <%s>\nConnecting...\r' % url)
+        sys.stdout.flush()
+        msg = ConfigOpener().retrieve(url, targetfile, reporthook=reporthook)
+        print ''  # clear the line
+        return targetfile
+    except:
+        print ' ** Error occurred while downloading\n <%s>' % url
+        raise
+
+def md5sum(targetfile):
+    """md5sum a file. Return the hex digest."""
+    digest = hashlib.md5()
+    with open(targetfile, 'rb') as f:
+      chunk = f.read(1024)
+      while chunk !=  "":
+        digest.update(chunk)
+        chunk = f.read(1024)
+    return digest.hexdigest()
+
+def unpack(packedfile, parent_path):
+    """Unpacks packedfile into parent_path. Assumes .zip. Returns parent_path"""
+    if zipfile.is_zipfile(packedfile):
+        with contextlib.closing(zipfile.ZipFile(packedfile, 'r')) as icuzip:
+            print ' Extracting zipfile: %s' % packedfile
+            icuzip.extractall(parent_path)
+            return parent_path
+    elif tarfile.is_tarfile(packedfile):
+        with tarfile.TarFile.open(packedfile, 'r') as icuzip:
+            print ' Extracting tarfile: %s' % packedfile
+            icuzip.extractall(parent_path)
+            return parent_path
+    else:
+        packedsuffix = packedfile.lower().split('.')[-1]  # .zip, .tgz etc
+        raise Exception('Error: Don\'t know how to unpack %s with extension %s' % (packedfile, packedsuffix))
+
+# List of possible "--download=" types.
+download_types = set(['icu'])
+
+# Default options for --download.
+download_default = "none"
+
+def help():
+  """This function calculates the '--help' text for '--download'."""
+  return """Select which packages may be auto-downloaded.
+valid values are: none, all, %s. (default is "%s").""" % (", ".join(download_types), download_default)
+
+def set2dict(keys, value=None):
+  """Convert some keys (iterable) to a dict."""
+  return dict((key, value) for (key) in keys)
+
+def parse(opt):
+  """This function parses the options to --download and returns a set such as { icu: true }, etc. """
+  if not opt:
+    opt = download_default
+
+  theOpts = set(opt.split(','))
+
+  if 'all' in theOpts:
+    # all on
+    return set2dict(download_types, True)
+  elif 'none' in theOpts:
+    # all off
+    return set2dict(download_types, False)
+
+  # OK. Now, process each of the opts.
+  theRet = set2dict(download_types, False)
+  for anOpt in opt.split(','):
+    if not anOpt or anOpt == "":
+      # ignore stray commas, etc.
+      continue
+    elif anOpt is 'all':
+      # all on
+      theRet = dict((key, True) for (key) in download_types)
+    else:
+      # turn this one on
+      if anOpt in download_types:
+        theRet[anOpt] = True
+      else:
+        # future proof: ignore unknown types
+        print 'Warning: ignoring unknown --download= type "%s"' % anOpt
+  # all done
+  return theRet
+
+def candownload(auto_downloads, package):
+  if not (package in auto_downloads.keys()):
+    raise Exception('Internal error: "%s" is not in the --downloads list. Check nodedownload.py' % package)
+  if auto_downloads[package]:
+    return True
+  else:
+    print """Warning: Not downloading package "%s". You could pass "--download=all"
+    (Windows: "download-all") to try auto-downloading it.""" % package
+    return False