Refactor decoder input stream
authorPetri Lehtinen <petri@digip.org>
Tue, 22 Feb 2011 08:47:02 +0000 (10:47 +0200)
committerPetri Lehtinen <petri@digip.org>
Tue, 22 Feb 2011 10:07:37 +0000 (12:07 +0200)
commit5df7b7939794abc9e8a826919d84d27385a6679d
tree73cd0ab2da897b98de04fee55a545a06321e74da
parente54ea1f7c904a5beef64ca0fe986cf7dc4f62296
Refactor decoder input stream

- Add a new field position to the json_error_t structure. This is the
  position in bytes from the beginning of the input.

- Keep track of line, column and input position in the stream level.
  Previously, only line was tracked, and it was in the lexer level, so
  this info was not available for UTF-8 decoding errors.

- While at it, refactor tests so that no separate "stripped" tests are
  required. json_process is now able to strip whitespace from its
  input, and the "valid" and "invalid" test suites now use this to
  test both non-stripped and stripped input.

Closes GH-9.
245 files changed:
doc/apiref.rst
src/error.c
src/jansson.h
src/jansson_private.h
src/load.c
src/pack_unpack.c
test/bin/json_process.c
test/suites/api/test_pack.c
test/suites/invalid-strip/apostrophe/error [deleted file]
test/suites/invalid-strip/apostrophe/input [deleted file]
test/suites/invalid-strip/ascii-unicode-identifier/error [deleted file]
test/suites/invalid-strip/ascii-unicode-identifier/input [deleted file]
test/suites/invalid-strip/brace-comma/error [deleted file]
test/suites/invalid-strip/brace-comma/input [deleted file]
test/suites/invalid-strip/bracket-comma/error [deleted file]
test/suites/invalid-strip/bracket-comma/input [deleted file]
test/suites/invalid-strip/bracket-one-comma/input [deleted file]
test/suites/invalid-strip/empty/error [deleted file]
test/suites/invalid-strip/empty/input [deleted file]
test/suites/invalid-strip/escaped-null-byte-in-string/error [deleted file]
test/suites/invalid-strip/escaped-null-byte-in-string/input [deleted file]
test/suites/invalid-strip/extra-comma-in-array/input [deleted file]
test/suites/invalid-strip/extra-command-in-multiline-array/error [deleted file]
test/suites/invalid-strip/extra-command-in-multiline-array/input [deleted file]
test/suites/invalid-strip/garbage-after-newline/error [deleted file]
test/suites/invalid-strip/garbage-after-newline/input [deleted file]
test/suites/invalid-strip/garbage-at-the-end/error [deleted file]
test/suites/invalid-strip/garbage-at-the-end/input [deleted file]
test/suites/invalid-strip/integer-starting-with-zero/error [deleted file]
test/suites/invalid-strip/integer-starting-with-zero/input [deleted file]
test/suites/invalid-strip/invalid-escape/error [deleted file]
test/suites/invalid-strip/invalid-escape/input [deleted file]
test/suites/invalid-strip/invalid-identifier/error [deleted file]
test/suites/invalid-strip/invalid-identifier/input [deleted file]
test/suites/invalid-strip/invalid-negative-integerr/input [deleted file]
test/suites/invalid-strip/invalid-negative-real/error [deleted file]
test/suites/invalid-strip/invalid-negative-real/input [deleted file]
test/suites/invalid-strip/invalid-second-surrogate/error [deleted file]
test/suites/invalid-strip/invalid-second-surrogate/input [deleted file]
test/suites/invalid-strip/lone-open-brace/input [deleted file]
test/suites/invalid-strip/lone-open-bracket/input [deleted file]
test/suites/invalid-strip/lone-second-surrogate/error [deleted file]
test/suites/invalid-strip/lone-second-surrogate/input [deleted file]
test/suites/invalid-strip/minus-sign-without-number/error [deleted file]
test/suites/invalid-strip/minus-sign-without-number/input [deleted file]
test/suites/invalid-strip/negative-integer-starting-with-zero/error [deleted file]
test/suites/invalid-strip/negative-integer-starting-with-zero/input [deleted file]
test/suites/invalid-strip/null-byte-in-string/error [deleted file]
test/suites/invalid-strip/null-byte-in-string/input [deleted file]
test/suites/invalid-strip/null-byte-outside-string/error [deleted file]
test/suites/invalid-strip/null-byte-outside-string/input [deleted file]
test/suites/invalid-strip/null/error [deleted file]
test/suites/invalid-strip/null/input [deleted file]
test/suites/invalid-strip/object-apostrophes/error [deleted file]
test/suites/invalid-strip/object-apostrophes/input [deleted file]
test/suites/invalid-strip/object-garbage-at-end/error [deleted file]
test/suites/invalid-strip/object-garbage-at-end/input [deleted file]
test/suites/invalid-strip/object-in-unterminated-array/input [deleted file]
test/suites/invalid-strip/object-no-colon/input [deleted file]
test/suites/invalid-strip/object-no-value/input [deleted file]
test/suites/invalid-strip/object-unterminated-value/input [deleted file]
test/suites/invalid-strip/real-garbage-after-e/error [deleted file]
test/suites/invalid-strip/real-garbage-after-e/input [deleted file]
test/suites/invalid-strip/real-negative-overflow/error [deleted file]
test/suites/invalid-strip/real-negative-overflow/input [deleted file]
test/suites/invalid-strip/real-positive-overflow/error [deleted file]
test/suites/invalid-strip/real-positive-overflow/input [deleted file]
test/suites/invalid-strip/real-truncated-at-e/error [deleted file]
test/suites/invalid-strip/real-truncated-at-e/input [deleted file]
test/suites/invalid-strip/real-truncated-at-point/error [deleted file]
test/suites/invalid-strip/real-truncated-at-point/input [deleted file]
test/suites/invalid-strip/run [deleted file]
test/suites/invalid-strip/tab-character-in-string/error [deleted file]
test/suites/invalid-strip/tab-character-in-string/input [deleted file]
test/suites/invalid-strip/too-big-negative-integer/error [deleted file]
test/suites/invalid-strip/too-big-negative-integer/input [deleted file]
test/suites/invalid-strip/too-big-positive-integer/error [deleted file]
test/suites/invalid-strip/too-big-positive-integer/input [deleted file]
test/suites/invalid-strip/truncated-unicode-surrogate/error [deleted file]
test/suites/invalid-strip/truncated-unicode-surrogate/input [deleted file]
test/suites/invalid-strip/unicode-identifier/error [deleted file]
test/suites/invalid-strip/unicode-identifier/input [deleted file]
test/suites/invalid-strip/unterminated-array-and-object/input [deleted file]
test/suites/invalid-strip/unterminated-array/input [deleted file]
test/suites/invalid-strip/unterminated-empty-key/input [deleted file]
test/suites/invalid-strip/unterminated-key/input [deleted file]
test/suites/invalid-strip/unterminated-object-and-array/error [deleted file]
test/suites/invalid-strip/unterminated-object-and-array/input [deleted file]
test/suites/invalid-strip/unterminated-string/input [deleted file]
test/suites/invalid-unicode/encoded-surrogate-half/error
test/suites/invalid-unicode/invalid-utf-8-after-backslash/error
test/suites/invalid-unicode/invalid-utf-8-in-array/error
test/suites/invalid-unicode/invalid-utf-8-in-bigger-int/error
test/suites/invalid-unicode/invalid-utf-8-in-escape/error
test/suites/invalid-unicode/invalid-utf-8-in-exponent/error
test/suites/invalid-unicode/invalid-utf-8-in-identifier/error
test/suites/invalid-unicode/invalid-utf-8-in-int/error
test/suites/invalid-unicode/invalid-utf-8-in-real-after-e/error
test/suites/invalid-unicode/invalid-utf-8-in-string/error
test/suites/invalid-unicode/lone-invalid-utf-8/error
test/suites/invalid-unicode/lone-utf-8-continuation-byte/error
test/suites/invalid-unicode/not-in-unicode-range/error
test/suites/invalid-unicode/overlong-3-byte-encoding/error
test/suites/invalid-unicode/overlong-4-byte-encoding/error
test/suites/invalid-unicode/overlong-ascii-encoding/error
test/suites/invalid-unicode/restricted-utf-8/error
test/suites/invalid-unicode/truncated-utf-8/error
test/suites/invalid/apostrophe/error
test/suites/invalid/ascii-unicode-identifier/error
test/suites/invalid/brace-comma/error
test/suites/invalid/bracket-comma/error
test/suites/invalid/bracket-one-comma/error [deleted file]
test/suites/invalid/bracket-one-comma/error.normal [moved from test/suites/invalid-strip/bracket-one-comma/error with 83% similarity]
test/suites/invalid/bracket-one-comma/error.strip [moved from test/suites/invalid-strip/lone-open-bracket/error with 83% similarity]
test/suites/invalid/empty/error
test/suites/invalid/escaped-null-byte-in-string/error
test/suites/invalid/extra-comma-in-array/error
test/suites/invalid/extra-comma-in-multiline-array/error [moved from test/suites/invalid-strip/extra-comma-in-array/error with 78% similarity]
test/suites/invalid/extra-comma-in-multiline-array/input [moved from test/suites/invalid/extra-command-in-multiline-array/input with 100% similarity]
test/suites/invalid/extra-command-in-multiline-array/error [deleted file]
test/suites/invalid/garbage-after-newline/error
test/suites/invalid/garbage-at-the-end/error
test/suites/invalid/integer-starting-with-zero/error
test/suites/invalid/invalid-escape/error
test/suites/invalid/invalid-identifier/error
test/suites/invalid/invalid-negative-integer/error [moved from test/suites/invalid-strip/invalid-negative-integerr/error with 80% similarity]
test/suites/invalid/invalid-negative-integer/input [moved from test/suites/invalid/invalid-negative-integerr/input with 100% similarity]
test/suites/invalid/invalid-negative-integerr/error [deleted file]
test/suites/invalid/invalid-negative-real/error
test/suites/invalid/invalid-second-surrogate/error
test/suites/invalid/lone-open-brace/error.normal [moved from test/suites/invalid-strip/lone-open-brace/error with 86% similarity]
test/suites/invalid/lone-open-brace/error.strip [moved from test/suites/invalid-strip/unterminated-array-and-object/error with 86% similarity]
test/suites/invalid/lone-open-bracket/error [deleted file]
test/suites/invalid/lone-open-bracket/error.normal [moved from test/suites/invalid-strip/object-in-unterminated-array/error with 83% similarity]
test/suites/invalid/lone-open-bracket/error.strip [moved from test/suites/invalid-strip/unterminated-array/error with 83% similarity]
test/suites/invalid/lone-second-surrogate/error
test/suites/invalid/minus-sign-without-number/error
test/suites/invalid/negative-integer-starting-with-zero/error
test/suites/invalid/null-byte-in-string/error
test/suites/invalid/null-byte-in-string/nostrip [new file with mode: 0644]
test/suites/invalid/null-byte-outside-string/error
test/suites/invalid/null-byte-outside-string/nostrip [new file with mode: 0644]
test/suites/invalid/null/error
test/suites/invalid/object-apostrophes/error
test/suites/invalid/object-garbage-at-end/error
test/suites/invalid/object-in-unterminated-array/error [deleted file]
test/suites/invalid/object-in-unterminated-array/error.normal [new file with mode: 0644]
test/suites/invalid/object-in-unterminated-array/error.strip [new file with mode: 0644]
test/suites/invalid/object-no-colon/error.normal [moved from test/suites/invalid-strip/object-no-colon/error with 83% similarity]
test/suites/invalid/object-no-colon/error.strip [moved from test/suites/invalid/object-no-colon/error with 83% similarity]
test/suites/invalid/object-no-value/error.normal [moved from test/suites/invalid-strip/object-no-value/error with 85% similarity]
test/suites/invalid/object-no-value/error.strip [moved from test/suites/invalid/object-no-value/error with 85% similarity]
test/suites/invalid/object-unterminated-value/error.normal [moved from test/suites/invalid/object-unterminated-value/error with 82% similarity]
test/suites/invalid/object-unterminated-value/error.strip [moved from test/suites/invalid-strip/object-unterminated-value/error with 84% similarity]
test/suites/invalid/real-garbage-after-e/error
test/suites/invalid/real-negative-overflow/error
test/suites/invalid/real-positive-overflow/error
test/suites/invalid/real-truncated-at-e/error
test/suites/invalid/real-truncated-at-point/error
test/suites/invalid/run
test/suites/invalid/tab-character-in-string/error
test/suites/invalid/too-big-negative-integer/error
test/suites/invalid/too-big-positive-integer/error
test/suites/invalid/truncated-unicode-surrogate/error
test/suites/invalid/unicode-identifier/error
test/suites/invalid/unterminated-array-and-object/error.normal [moved from test/suites/invalid/lone-open-brace/error with 86% similarity]
test/suites/invalid/unterminated-array-and-object/error.strip [moved from test/suites/invalid/unterminated-array-and-object/error with 86% similarity]
test/suites/invalid/unterminated-array/error [deleted file]
test/suites/invalid/unterminated-array/error.normal [new file with mode: 0644]
test/suites/invalid/unterminated-array/error.strip [new file with mode: 0644]
test/suites/invalid/unterminated-empty-key/error.normal [moved from test/suites/invalid/unterminated-empty-key/error with 82% similarity]
test/suites/invalid/unterminated-empty-key/error.strip [moved from test/suites/invalid-strip/unterminated-empty-key/error with 84% similarity]
test/suites/invalid/unterminated-key/error.normal [moved from test/suites/invalid/unterminated-key/error with 82% similarity]
test/suites/invalid/unterminated-key/error.strip [moved from test/suites/invalid-strip/unterminated-key/error with 84% similarity]
test/suites/invalid/unterminated-object-and-array/error
test/suites/invalid/unterminated-string/error.normal [moved from test/suites/invalid/unterminated-string/error with 82% similarity]
test/suites/invalid/unterminated-string/error.strip [moved from test/suites/invalid-strip/unterminated-string/error with 84% similarity]
test/suites/valid-strip/complex-array/input [deleted file]
test/suites/valid-strip/complex-array/output [deleted file]
test/suites/valid-strip/empty-array/input [deleted file]
test/suites/valid-strip/empty-array/output [deleted file]
test/suites/valid-strip/empty-object-in-array/input [deleted file]
test/suites/valid-strip/empty-object-in-array/output [deleted file]
test/suites/valid-strip/empty-object/input [deleted file]
test/suites/valid-strip/empty-object/output [deleted file]
test/suites/valid-strip/empty-string/input [deleted file]
test/suites/valid-strip/empty-string/output [deleted file]
test/suites/valid-strip/escaped-utf-control-char/input [deleted file]
test/suites/valid-strip/escaped-utf-control-char/output [deleted file]
test/suites/valid-strip/false/input [deleted file]
test/suites/valid-strip/false/output [deleted file]
test/suites/valid-strip/negative-int/input [deleted file]
test/suites/valid-strip/negative-int/output [deleted file]
test/suites/valid-strip/negative-one/input [deleted file]
test/suites/valid-strip/negative-one/output [deleted file]
test/suites/valid-strip/negative-zero/input [deleted file]
test/suites/valid-strip/negative-zero/output [deleted file]
test/suites/valid-strip/null/input [deleted file]
test/suites/valid-strip/null/output [deleted file]
test/suites/valid-strip/one-byte-utf-8/input [deleted file]
test/suites/valid-strip/one-byte-utf-8/output [deleted file]
test/suites/valid-strip/real-capital-e-negative-exponent/input [deleted file]
test/suites/valid-strip/real-capital-e-negative-exponent/output [deleted file]
test/suites/valid-strip/real-capital-e-positive-exponent/input [deleted file]
test/suites/valid-strip/real-capital-e-positive-exponent/output [deleted file]
test/suites/valid-strip/real-capital-e/input [deleted file]
test/suites/valid-strip/real-capital-e/output [deleted file]
test/suites/valid-strip/real-exponent/input [deleted file]
test/suites/valid-strip/real-exponent/output [deleted file]
test/suites/valid-strip/real-fraction-exponent/input [deleted file]
test/suites/valid-strip/real-fraction-exponent/output [deleted file]
test/suites/valid-strip/real-negative-exponent/input [deleted file]
test/suites/valid-strip/real-negative-exponent/output [deleted file]
test/suites/valid-strip/real-positive-exponent/input [deleted file]
test/suites/valid-strip/real-positive-exponent/output [deleted file]
test/suites/valid-strip/real-underflow/input [deleted file]
test/suites/valid-strip/real-underflow/output [deleted file]
test/suites/valid-strip/run [deleted file]
test/suites/valid-strip/short-string/input [deleted file]
test/suites/valid-strip/short-string/output [deleted file]
test/suites/valid-strip/simple-ascii-string/input [deleted file]
test/suites/valid-strip/simple-ascii-string/output [deleted file]
test/suites/valid-strip/simple-int-0/input [deleted file]
test/suites/valid-strip/simple-int-0/output [deleted file]
test/suites/valid-strip/simple-int-1/input [deleted file]
test/suites/valid-strip/simple-int-1/output [deleted file]
test/suites/valid-strip/simple-int-123/input [deleted file]
test/suites/valid-strip/simple-int-123/output [deleted file]
test/suites/valid-strip/simple-object/input [deleted file]
test/suites/valid-strip/simple-object/output [deleted file]
test/suites/valid-strip/simple-real/input [deleted file]
test/suites/valid-strip/simple-real/output [deleted file]
test/suites/valid-strip/string-escapes/input [deleted file]
test/suites/valid-strip/string-escapes/output [deleted file]
test/suites/valid-strip/three-byte-utf-8/input [deleted file]
test/suites/valid-strip/three-byte-utf-8/output [deleted file]
test/suites/valid-strip/true/input [deleted file]
test/suites/valid-strip/true/output [deleted file]
test/suites/valid-strip/two-byte-utf-8/input [deleted file]
test/suites/valid-strip/two-byte-utf-8/output [deleted file]
test/suites/valid-strip/utf-8-string/input [deleted file]
test/suites/valid-strip/utf-8-string/output [deleted file]
test/suites/valid-strip/utf-surrogate-four-byte-encoding/input [deleted file]
test/suites/valid-strip/utf-surrogate-four-byte-encoding/output [deleted file]
test/suites/valid/run