summaryrefslogtreecommitdiff
path: root/lib/string_decoder.js
AgeCommit message (Collapse)Author
2018-10-04bootstrapper: move internalBinding to NativeModuleGus Caplan
internalBinding is used so often that it should just automatically be available for usage in internals. PR-URL: https://github.com/nodejs/node/pull/23025 Refs: https://github.com/nodejs/node/commit/2a9eb31 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Michaël Zasso <targos@protonmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com> Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com> Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
2018-09-17string_decoder: support typed array or data viewBenjamin Chen
Refs: https://github.com/nodejs/node/issues/1826 PR-URL: https://github.com/nodejs/node/pull/22562 Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
2018-03-25lib: always show ERR_INVALID_ARG_TYPE received partRuben Bridgewater
This makes a effort to make sure all of these errors will actually also show the received input. On top of that it refactors a few tests for better maintainability. It will also change the returned type to always be a simple typeof instead of special handling null. PR-URL: https://github.com/nodejs/node/pull/19445 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net>
2018-03-15src: put bootstrappers in lib/internal/bootstrap/Joyee Cheung
Create `lib/internal/bootstrap/` and put bootstrappers there: Before: ``` lib/internal ├── ... ├── bootstrap_loaders.js └── bootstrap_node.js ``` After: ``` lib/internal ├── ... └── bootstrap ├── loaders.js └── node.js ``` PR-URL: https://github.com/nodejs/node/pull/19177 Refs: https://github.com/nodejs/node/pull/19112 Reviewed-By: Gus Caplan <me@gus.host> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
2018-03-06src: move internal loaders out of bootstrap_node.jsJoyee Cheung
- Moves the creation of `process.binding()`, `process._linkedBinding()` `internalBinding()` and `NativeModule` into a separate file `lib/internal/bootstrap_loaders.js`, and documents them there. This file will be compiled and run before `bootstrap_node.js`, which means we now bootstrap the internal module & binding system before actually bootstrapping Node.js. - Rename the special ID that can be used to require `NativeModule` as `internal/bootstrap_loaders` since it is setup there. Also put `internalBinding` in the object exported by `NativeModule.require` instead of putting it inside the `NativeModule.wrapper` - Use the original `getBinding()` to get the source code of native modules instead of getting it from `process.binding('native')` so that users cannot fake native modules by modifying the binding object. - Names the bootstrapping functions so their names show up in the stack trace. PR-URL: https://github.com/nodejs/node/pull/19112 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Anatoli Papirovski <apapirovski@mac.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Gus Caplan <me@gus.host>
2018-03-05lib: port errors to new systemMichaël Zasso
This is a first batch of updates that touches non-underscored modules in lib. PR-URL: https://github.com/nodejs/node/pull/19034 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
2018-03-02lib: improve normalize encoding performanceRuben Bridgewater
This focuses on the common case by making sure they are prioritized. It also changes some typeof checks to test for undefined since that is faster and it adds a benchmark. PR-URL: https://github.com/nodejs/node/pull/18790 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
2018-02-13string_decoder: fix regressionsAnatoli Papirovski
There are libraries which invoke StringDecoder using .call and .inherits, which directly conflicts with making StringDecoder be a class which can only be invoked with the new keyword. Revert to declaring it as a function. StringDecoder#lastNeed was not defined, redefine it using the new interface and fix StringDecoder#lastTotal. PR-URL: https://github.com/nodejs/node/pull/18723 Refs: https://github.com/nodejs/node/pull/18537 Reviewed-By: Michaël Zasso <targos@protonmail.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Evan Lucas <evanlucas@me.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
2018-02-10string_decoder: reimplement in C++Anna Henningsen
Implement string decoder in C++. The perks are a decent speed boost (for decoding, whereas creation show some performance degradation), that this can now be used more easily to add native decoding support to C++ streams and (arguably) more readable variable names. PR-URL: https://github.com/nodejs/node/pull/18537 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
2018-02-02string_decoder: reset decoder on endJustin Ridgewell
This resets the StringDecoder's state after calling `#end`. Further writes to the decoder will act as if it were a brand new instance, allowing simple reuse. PR-URL: https://github.com/nodejs/node/pull/18494 Fixes: https://github.com/nodejs/node/issues/16564 Refs: https://github.com/nodejs/node/pull/16594 Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com> Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: Anna Henningsen <anna@addaleax.net>
2017-10-16lib: use destructuring for some constantsWeijia Wang
This change is to unify the declaration for constants into using destructuring on the top-level-module scope, reducing some redundant code. PR-URL: https://github.com/nodejs/node/pull/16063 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Tobias Nießen <tniessen@tnie.de> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
2017-09-13string_decoder: Migrate to use internal/errorsWeijia Wang
PR-URL: https://github.com/nodejs/node/pull/14682 Refs: https://github.com/nodejs/node/issues/11273 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Michaël Zasso <targos@protonmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
2017-03-10meta: restore original copyright headerJames M Snell
A prior io.js era commit inappropriately removed the original copyright statements from the source. This restores those in any files still remaining from that edit. Ref: https://github.com/nodejs/TSC/issues/174 Ref: https://github.com/nodejs/node/pull/10599 PR-URL: https://github.com/nodejs/node/pull/10155 Note: This PR was required, reviewed-by and approved by the Node.js Foundation Legal Committee and the TSC. There is no `Approved-By:` meta data.
2017-01-26string_decoder: align UTF-8 handling with V8Brian White
V8 5.5 changed how invalid characters are handled and it now appears to follow the WHATWG Encoding standard, where all of an invalid character's bytes are replaced by a single replacement character (\ufffd) instead of replacing each invalid byte with separate replacement characters. Example: the byte sequence 0xF0,0xB8,0x41 is decoded as '\ufffdA' in V8 5.5, but is decoded as '\ufffd\ufffdA' in previous versions of V8. PR-URL: https://github.com/nodejs/node/pull/9618 Reviewed-By: Ali Ijaz Sheikh <ofrobots@google.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
2016-06-23string_decoder: fix bad utf8 character handlingBrian White
This commit fixes an issue when extra utf8 continuation bytes appear at the end of a chunk of data, causing miscalculations to be made when checking how many bytes are needed to decode a complete character. Fixes: https://github.com/nodejs/node/issues/7308 PR-URL: https://github.com/nodejs/node/pull/7310 Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Fedor Indutny <fedor.indutny@gmail.com>
2016-06-21buffer,string_decoder: consolidate encoding validation logicJames M Snell
Buffer.isEncoding and string_decoder.normalizeEncoding shared quite a bit of logic. This moves the primary logic into internal/util. The userland modules that monkey patch Buffer.isEncoding should still work. PR-URL: https://github.com/nodejs/node/pull/7207 Reviewed-By: Brian White <mscdex@mscdex.net> Reviewed-By: Trevor Norris <trev.norris@gmail.com>
2016-06-07buffer: introduce latin1 encoding termTrevor Norris
When node began using the OneByte API (f150d56) it also switched to officially supporting ISO-8859-1. Though at the time no new encoding string was introduced. Introduce the new encoding string 'latin1' to be more explicit. The previous 'binary' and documented as an alias to 'latin1'. While many tests have switched to use 'latin1', there are still plenty that do both 'binary' and 'latin1' checks side-by-side to ensure there is no regression. PR-URL: https://github.com/nodejs/node/pull/7111 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: James M Snell <jasnell@gmail.com>
2016-05-29string_decoder: rewrite implementationBrian White
This commit provides a rewrite of StringDecoder that both improves performance (for non-single-byte encodings) and understandability. Additionally, StringDecoder instantiation performance has increased considerably due to inlinability and more efficient encoding name checking. PR-URL: https://github.com/nodejs/node/pull/6777 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
2016-04-27doc: use Buffer.from() instead of new Buffer()Jackson Tian
Use new API of Buffer to developers in most documents. PR-URL: https://github.com/nodejs/node/pull/6367 Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Сковорода Никита Андреевич <chalkerx@gmail.com>
2016-03-16buffer: add .from(), .alloc() and .allocUnsafe()James M Snell
Several changes: * Soft-Deprecate Buffer() constructors * Add `Buffer.from()`, `Buffer.alloc()`, and `Buffer.allocUnsafe()` * Add `--zero-fill-buffers` command line option * Add byteOffset and length to `new Buffer(arrayBuffer)` constructor * buffer.fill('') previously had no effect, now zero-fills * Update the docs PR-URL: https://github.com/nodejs/node/pull/4682 Reviewed-By: Сковорода Никита Андреевич <chalkerx@gmail.com> Reviewed-By: Stephen Belanger <admin@stephenbelanger.com>
2016-02-11string_decoder: fix performance regressionBrian White
This commit reverts the const usage introduced by 68a6abc because v8 currently cannot optimize functions that contain these uses of const (unsupported phi use of const variable). The performance difference in this case can be up to ~130% for non-ascii/binary string encodings. PR-URL: https://github.com/nodejs/node/pull/5134 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: James M Snell <jasnell@gmail.com>
2016-02-01lib: remove string_decoder.js var redeclarationsRich Trott
PR-URL: https://github.com/nodejs/node/pull/4978 Reviewed-By: Brian White <mscdex@mscdex.net> Reviewed-By: James M Snell <jasnell@gmail.com>
2015-06-11lib: don't use global BufferRoman Reiss
Port of https://github.com/joyent/node/pull/8603 The race condition present in the original PR didn't occur, so no workaround was needed. PR-URL: https://github.com/nodejs/io.js/pull/1794 Reviewed-By: Trevor Norris <trev.norris@gmail.com>
2015-04-28string_decoder: don't cache Buffer.isEncodingBrian White
Some modules are monkey-patching Buffer.isEncoding, so without this they cannot do that. Fixes: https://github.com/iojs/io.js/issues/1547 PR-URL: https://github.com/iojs/io.js/pull/1548 Reviewed-By: Evan Lucas <evanlucas@me.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
2015-03-25string_decoder: optimize write()Brian White
By limiting property getting/setting to only where they are absolutely necessary, we can achieve greater performance especially with small utf8 inputs and any size base64 inputs. PR-URL: https://github.com/iojs/io.js/pull/1209 Reviewed-By: Rod Vagg <rod@vagg.org> Reviewed-By: Nicu Micleușanu <micnic90@gmail.com> Reviewed-By: Chris Dickinson <christopher.s.dickinson@gmail.com>
2015-01-21lib: use const to define constantscjihrig
This commit replaces a number of var statements throughout the lib code with const statements. PR-URL: https://github.com/iojs/io.js/pull/541 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
2015-01-12Remove excessive copyright/license boilerplateisaacs
The copyright and license notice is already in the LICENSE file. There is no justifiable reason to also require that it be included in every file, since the individual files are not individually distributed except as part of the entire package.
2014-11-22lib: turn on strict modeBen Noordhuis
Turn on strict mode for the files in the lib/ directory. It helps catch bugs and can have a positive effect on performance. PR-URL: https://github.com/node-forward/node/pull/64 Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Fedor Indutny <fedor@indutny.com>
2014-07-15lib: jslint string_decoder.jsFedor Indutny
2014-06-06string_decoder: Add more commentsFelix Geisendörfer
2014-06-06string_decoder: Fix failures from new test casesFelix Geisendörfer
This patch simplifies the implementation of StringDecoder, fixes the failures from the new test cases, and also no longer relies on v8's WriteUtf8 function to encode individual surrogates.
2012-12-13Don't allow invalid encodings in StringDecoder classisaacs
2012-10-11string_decoder: Add 'end' method, do base64 properlyisaacs
2012-05-05string_decoder: added support for UTF-16LEkoichik
Fixes #3223.
2012-05-05string_decoder: add support for CESU-8koichik
Fixes #3217.
2011-03-14Update copyright headersRyan Dahl
2010-12-04Fix global leaksRyan Dahl
2010-12-01lintRyan Dahl
2010-09-28Remove require('buffer') in built-in libraries.Ryan Dahl
2010-06-29fix whitespace errorsBlake Mizerany
2010-06-29Revert "Buffer.copy should copy through sourceEnd, as specified."Ryan Dahl
This reverts commit a2f70da4c9bf14a6e0c4a17cac11a5ab555ce90f. Keep tests modifies a few edge checks on Copy()
2010-06-25Buffer.copy should copy through sourceEnd, as specified.Matt Ranney
Improve test-buffer.js to cover all copy error cases. Fix off by one error in string_decoder.
2010-06-15Refactor: Utf8Decoder -> StringDecoderRyan Dahl
Instead of just decoding Utf8, this will proxy requests to buffer.toString() for other encodings. This makes for a simpler interface.
generated by cgit v1.2.3 (git 2.39.1) at 2024-04-27 19:30:30 +0000