1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
|
Overview [](https://travis-ci.org/lydell/js-tokens)
========
A regex that tokenizes JavaScript.
```js
var jsTokens = require("js-tokens").default
var jsString = "var foo=opts.foo;\n..."
jsString.match(jsTokens)
// ["var", " ", "foo", "=", "opts", ".", "foo", ";", "\n", ...]
```
Installation
============
`npm install js-tokens`
```js
import jsTokens from "js-tokens"
// or:
var jsTokens = require("js-tokens").default
```
Usage
=====
### `jsTokens` ###
A regex with the `g` flag that matches JavaScript tokens.
The regex _always_ matches, even invalid JavaScript and the empty string.
The next match is always directly after the previous.
### `var token = matchToToken(match)` ###
```js
import {matchToToken} from "js-tokens"
// or:
var matchToToken = require("js-tokens").matchToToken
```
Takes a `match` returned by `jsTokens.exec(string)`, and returns a `{type:
String, value: String}` object. The following types are available:
- string
- comment
- regex
- number
- name
- punctuator
- whitespace
- invalid
Multi-line comments and strings also have a `closed` property indicating if the
token was closed or not (see below).
Comments and strings both come in several flavors. To distinguish them, check if
the token starts with `//`, `/*`, `'`, `"` or `` ` ``.
Names are ECMAScript IdentifierNames, that is, including both identifiers and
keywords. You may use [is-keyword-js] to tell them apart.
Whitespace includes both line terminators and other whitespace.
[is-keyword-js]: https://github.com/crissdev/is-keyword-js
ECMAScript support
==================
The intention is to always support the latest stable ECMAScript version.
If adding support for a newer version requires changes, a new version with a
major verion bump will be released.
Currently, [ECMAScript 2017] is supported.
[ECMAScript 2017]: https://www.ecma-international.org/ecma-262/8.0/index.html
Invalid code handling
=====================
Unterminated strings are still matched as strings. JavaScript strings cannot
contain (unescaped) newlines, so unterminated strings simply end at the end of
the line. Unterminated template strings can contain unescaped newlines, though,
so they go on to the end of input.
Unterminated multi-line comments are also still matched as comments. They
simply go on to the end of the input.
Unterminated regex literals are likely matched as division and whatever is
inside the regex.
Invalid ASCII characters have their own capturing group.
Invalid non-ASCII characters are treated as names, to simplify the matching of
names (except unicode spaces which are treated as whitespace).
Regex literals may contain invalid regex syntax. They are still matched as
regex literals. They may also contain repeated regex flags, to keep the regex
simple.
Strings may contain invalid escape sequences.
Limitations
===========
Tokenizing JavaScript using regexes—in fact, _one single regex_—won’t be
perfect. But that’s not the point either.
You may compare jsTokens with [esprima] by using `esprima-compare.js`.
See `npm run esprima-compare`!
[esprima]: http://esprima.org/
### Template string interpolation ###
Template strings are matched as single tokens, from the starting `` ` `` to the
ending `` ` ``, including interpolations (whose tokens are not matched
individually).
Matching template string interpolations requires recursive balancing of `{` and
`}`—something that JavaScript regexes cannot do. Only one level of nesting is
supported.
### Division and regex literals collision ###
Consider this example:
```js
var g = 9.82
var number = bar / 2/g
var regex = / 2/g
```
A human can easily understand that in the `number` line we’re dealing with
division, and in the `regex` line we’re dealing with a regex literal. How come?
Because humans can look at the whole code to put the `/` characters in context.
A JavaScript regex cannot. It only sees forwards.
When the `jsTokens` regex scans throught the above, it will see the following
at the end of both the `number` and `regex` rows:
```js
/ 2/g
```
It is then impossible to know if that is a regex literal, or part of an
expression dealing with division.
Here is a similar case:
```js
foo /= 2/g
foo(/= 2/g)
```
The first line divides the `foo` variable with `2/g`. The second line calls the
`foo` function with the regex literal `/= 2/g`. Again, since `jsTokens` only
sees forwards, it cannot tell the two cases apart.
There are some cases where we _can_ tell division and regex literals apart,
though.
First off, we have the simple cases where there’s only one slash in the line:
```js
var foo = 2/g
foo /= 2
```
Regex literals cannot contain newlines, so the above cases are correctly
identified as division. Things are only problematic when there are more than
one non-comment slash in a single line.
Secondly, not every character is a valid regex flag.
```js
var number = bar / 2/e
```
The above example is also correctly identified as division, because `e` is not a
valid regex flag. I initially wanted to future-proof by allowing `[a-zA-Z]*`
(any letter) as flags, but it is not worth it since it increases the amount of
ambigous cases. So only the standard `g`, `m`, `i`, `y` and `u` flags are
allowed. This means that the above example will be identified as division as
long as you don’t rename the `e` variable to some permutation of `gmiyu` 1 to 5
characters long.
Lastly, we can look _forward_ for information.
- If the token following what looks like a regex literal is not valid after a
regex literal, but is valid in a division expression, then the regex literal
is treated as division instead. For example, a flagless regex cannot be
followed by a string, number or name, but all of those three can be the
denominator of a division.
- Generally, if what looks like a regex literal is followed by an operator, the
regex literal is treated as division instead. This is because regexes are
seldomly used with operators (such as `+`, `*`, `&&` and `==`), but division
could likely be part of such an expression.
Please consult the regex source and the test cases for precise information on
when regex or division is matched (should you need to know). In short, you
could sum it up as:
If the end of a statement looks like a regex literal (even if it isn’t), it
will be treated as one. Otherwise it should work as expected (if you write sane
code).
License
=======
[MIT](LICENSE).
|
