Support languageservices expressions test data by DarkaMaul · Pull Request #1691 · zizmorcore/zizmor

DarkaMaul · 2026-03-04T13:43:40Z

Pre-submission checks

Please check these boxes:

Mandatory: This PR corresponds to an issue (if not, please create
one first) (Tests: integrate GitHub's official expression tests #1688)
I hereby disclose the use of an LLM or other AI coding assistant in the
creation of this PR. PRs will not be rejected for using AI tools, but
will be rejected for undisclosed use.

If a checkbox is not applicable, you can leave it unchecked.

Summary

This PR adds a mechanism to support the KAT from actions/languageservices .

Test Plan

This is a test only PR.

⚠️ This PR should be merged as if because it contains known failure! ⚠️

DarkaMaul · 2026-03-04T13:51:16Z

The current test result is:

     Running tests/upstream_kat.rs (target/debug/deps/upstream_kat-5564006d1e6407e3)

running 1 test
test test_upstream_kat ... FAILED

failures:

---- test_upstream_kat stdout ----

thread 'test_upstream_kat' (146517) panicked at crates/github-actions-expressions/tests/upstream_kat.rs:178:5:
122 upstream KAT failure(s):
   1. basic.json::unknown context[0] `nosuchcontext.foo`: expected parsing error "Unrecognized named-value: 'nosuchcontext'. Located at position 1 within expression: nosuchcontext.foo" but parsed OK
   2. basic.json::empty_expression[0] ``: expected result but parse failed:  --> 1:1
  |
1 | 
  | ^---
  |
  = expected unary_expr
   3. coerce_number.json::object[0] `fromjson('[]') != NaN`: expected Boolean(true), but consteval returned None
   4. coerce_number.json::object[1] `fromjson('[]') == NaN`: expected Boolean(false), but consteval returned None
   5. coerce_number.json::object[2] `fromjson('[]') > NaN`: expected Boolean(false), but consteval returned None
   6. coerce_number.json::object[3] `fromjson('[]') < NaN`: expected Boolean(false), but consteval returned None
   7. coerce_number.json::string[7] `' Infinity ' == Infinity`: expected Boolean(true), but consteval returned None
   8. coerce_number.json::string[8] `' -Infinity ' == -Infinity`: expected Boolean(true), but consteval returned None
   9. coerce_number.json::string[9] `' NaN ' != NaN`: expected Boolean(true), but consteval returned None
  10. coerce_number.json::string[10] `' NaN ' == NaN`: expected Boolean(false), but consteval returned None
  11. coerce_number.json::string[11] `' NaN ' > NaN`: expected Boolean(false), but consteval returned None
  12. coerce_number.json::string[12] `' NaN ' < NaN`: expected Boolean(false), but consteval returned None
  13. coerce_number.json::string[13] `' abc ' != NaN`: expected Boolean(true), but consteval returned None
  14. coerce_number.json::string[14] `' abc ' == NaN`: expected Boolean(false), but consteval returned None
  15. coerce_number.json::string[15] `' abc ' > NaN`: expected Boolean(false), but consteval returned None
  16. coerce_number.json::string[16] `' abc ' < NaN`: expected Boolean(false), but consteval returned None
  17. coerce_number.json::array[0] `fromjson('[]') != NaN`: expected Boolean(true), but consteval returned None
  18. coerce_number.json::array[1] `fromjson('[]') == NaN`: expected Boolean(false), but consteval returned None
  19. coerce_number.json::array[2] `fromjson('[]') > NaN`: expected Boolean(false), but consteval returned None
  20. coerce_number.json::array[3] `fromjson('[]') < NaN`: expected Boolean(false), but consteval returned None
  21. coerce_string.json::number[15] `format('{0}', 0.84551240822557006)`: expected String("0.84551240822557"), got String("0.8455124082255701")
  22. coerce_string.json::number[27] `format('{0}', Infinity)`: expected String("Infinity"), but consteval returned None
  23. coerce_string.json::number[28] `format('{0}', -Infinity)`: expected String("-Infinity"), but consteval returned None
  24. coerce_string.json::number[29] `format('{0}', NaN)`: expected String("NaN"), but consteval returned None
  25. format.json::format[16] `format('{0}', Infinity)`: expected String("Infinity"), but consteval returned None
  26. format.json::format[17] `format('{0}', -Infinity)`: expected String("-Infinity"), but consteval returned None
  27. format.json::format[18] `format('{0}', NaN)`: expected String("NaN"), but consteval returned None
  28. fromJSON.json::array[0] `fromJSON('[]')`: unknown result kind "Array"
  29. fromJSON.json::array[1] `fromJSON('[1, 2, 3]')`: unknown result kind "Array"
  30. fromJSON.json::array[2] `fromJSON('[[1, 2, 3], ["abc","def","ghi"], [true, false, null, [], {}]]')`: unknown result kind "Array"
  31. fromJSON.json::object[0] `fromJSON('{}')`: unknown result kind "Object"
  32. fromJSON.json::object[1] `fromJSON('{"one": "value one", "two": "value two", "three": "value three"}')`: unknown result kind "Object"
  33. fromJSON.json::object[2] `fromJSON('{"nested-one": {"one": 1,"two": 2,"three": 3},"nested-two": {"string one": "value one","string two": "value two","string three": "value three"},"nested-three": {"true": true,"false": false,"null": null,"array": [],"object": {}}
}')`: unknown result kind "Object"
  34. join.json::join[15] `join()`: expected parsing error "Too few parameters supplied: " but parsed OK
  35. join.json::join[16] `join(1, 2, 3)`: expected parsing error "Too many parameters supplied: " but parsed OK
  36. number.json::number[8] `format('{0}', -Infinity)`: expected String("-Infinity"), but consteval returned None
  37. number.json::number[9] `format('{0}', Infinity)`: expected String("Infinity"), but consteval returned None
  38. number.json::number[10] `format('{0}', +Infinity)`: expected result but parse failed:  --> 1:15
  |
1 | format('{0}', +Infinity)
  |               ^---
  |
  = expected unary_expr
  39. number.json::number[11] `format('{0}', NaN)`: expected String("NaN"), but consteval returned None
  40. number.json::number[32] `-Inf`: expected lexing error "Unexpected symbol: '-Inf'. Located at position 1 within expression: -Inf" but parsed OK
  41. op_dot.json::property-basics[7] `fromJson('{"one": "one val"}').one`: expected String("one val"), but consteval returned None
  42. op_dot.json::property-basics[8] `(fromJson('{"one": "one val"}')).one`: expected result but parse failed:  --> 1:33
  |
1 | (fromJson('{"one": "one val"}')).one
  |                                 ^---
  |
  = expected EOI, eq_op, or comp_op
  43. op_eq.json::coerce_number_bool[2] `NaN == false`: expected Boolean(false), but consteval returned None
  44. op_eq.json::coerce_string_number[11] `' Infinity ' == Infinity`: expected Boolean(true), but consteval returned None
  45. op_eq.json::coerce_string_number[12] `' -Infinity ' == -Infinity`: expected Boolean(true), but consteval returned None
  46. op_eq.json::coerce_string_number[13] `'NaN' == NaN`: expected Boolean(false), but consteval returned None
  47. op_eq.json::coerce_number_string[11] `Infinity == ' Infinity '`: expected Boolean(true), but consteval returned None
  48. op_eq.json::coerce_number_string[12] `-Infinity == ' -Infinity '`: expected Boolean(true), but consteval returned None
  49. op_eq.json::coerce_number_string[13] `NaN == 'NaN'`: expected Boolean(false), but consteval returned None
  50. op_eq.json::coerce_bool_number[2] `false == NaN`: expected Boolean(false), but consteval returned None
  51. op_eq.json::number[5] `NaN == NaN`: expected Boolean(false), but consteval returned None
  52. op_eq.json::number[6] `Infinity == Infinity`: expected Boolean(true), but consteval returned None
  53. op_eq.json::number[7] `-Infinity == Infinity`: expected Boolean(false), but consteval returned None
  54. op_eq.json::number[8] `0 == Infinity`: expected Boolean(false), but consteval returned None
  55. op_eq.json::number[9] `1 == Infinity`: expected Boolean(false), but consteval returned None
  56. op_gt.json::coerce_number_bool[4] `NaN > false`: expected Boolean(false), but consteval returned None
  57. op_gt.json::coerce_number_bool[5] `NaN > true`: expected Boolean(false), but consteval returned None
  58. op_gt.json::coerce_number_bool[6] `Infinity > false`: expected Boolean(true), but consteval returned None
  59. op_gt.json::coerce_number_bool[7] `Infinity > true`: expected Boolean(true), but consteval returned None
  60. op_gt.json::coerce_number_bool[8] `-Infinity > false`: expected Boolean(false), but consteval returned None
  61. op_gt.json::coerce_number_bool[9] `-Infinity > true`: expected Boolean(false), but consteval returned None
  62. op_gt.json::coerce_bool_number[4] `false > NaN`: expected Boolean(false), but consteval returned None
  63. op_gt.json::coerce_bool_number[5] `true > NaN`: expected Boolean(false), but consteval returned None
  64. op_gt.json::coerce_bool_number[6] `false > Infinity`: expected Boolean(false), but consteval returned None
  65. op_gt.json::coerce_bool_number[7] `true > Infinity`: expected Boolean(false), but consteval returned None
  66. op_gt.json::coerce_bool_number[8] `false > -Infinity`: expected Boolean(true), but consteval returned None
  67. op_gt.json::coerce_bool_number[9] `true > -Infinity`: expected Boolean(true), but consteval returned None
  68. op_idx.json::index-following-function[0] `fromJson('["one", "two"]')[1]`: expected String("two"), but consteval returned None
  69. op_idx.json::index-following-group[0] `(fromJson('["one", "two"]'))[1]`: expected result but parse failed:  --> 1:29
  |
1 | (fromJson('["one", "two"]'))[1]
  |                             ^---
  |
  = expected EOI, eq_op, or comp_op
  70. op_lt.json::coerce_number_bool[4] `NaN < false`: expected Boolean(false), but consteval returned None
  71. op_lt.json::coerce_number_bool[5] `NaN < true`: expected Boolean(false), but consteval returned None
  72. op_lt.json::coerce_number_bool[6] `Infinity < false`: expected Boolean(false), but consteval returned None
  73. op_lt.json::coerce_number_bool[7] `Infinity < true`: expected Boolean(false), but consteval returned None
  74. op_lt.json::coerce_number_bool[8] `-Infinity < false`: expected Boolean(true), but consteval returned None
  75. op_lt.json::coerce_number_bool[9] `-Infinity < true`: expected Boolean(true), but consteval returned None
  76. op_lt.json::coerce_bool_number[4] `false < NaN`: expected Boolean(false), but consteval returned None
  77. op_lt.json::coerce_bool_number[5] `true < NaN`: expected Boolean(false), but consteval returned None
  78. op_lt.json::coerce_bool_number[6] `false < Infinity`: expected Boolean(true), but consteval returned None
  79. op_lt.json::coerce_bool_number[7] `true < Infinity`: expected Boolean(true), but consteval returned None
  80. op_lt.json::coerce_bool_number[8] `false < -Infinity`: expected Boolean(false), but consteval returned None
  81. op_lt.json::coerce_bool_number[9] `true < -Infinity`: expected Boolean(false), but consteval returned None
  82. op_ne.json::coerce_number_bool[4] `NaN != false`: expected Boolean(true), but consteval returned None
  83. op_ne.json::coerce_number_string[11] `Infinity != ' Infinity '`: expected Boolean(false), but consteval returned None
  84. op_ne.json::coerce_number_string[12] `-Infinity != ' -Infinity '`: expected Boolean(false), but consteval returned None
  85. op_ne.json::coerce_number_string[13] `NaN != 'NaN'`: expected Boolean(true), but consteval returned None
  86. op_ne.json::number[5] `NaN != NaN`: expected Boolean(true), but consteval returned None
  87. op_ne.json::number[6] `0 != NaN`: expected Boolean(true), but consteval returned None
  88. op_ne.json::number[7] `1 != NaN`: expected Boolean(true), but consteval returned None
  89. op_ne.json::number[8] `Infinity != Infinity`: expected Boolean(false), but consteval returned None
  90. op_ne.json::number[9] `-Infinity != Infinity`: expected Boolean(true), but consteval returned None
  91. op_ne.json::number[10] `0 != Infinity`: expected Boolean(true), but consteval returned None
  92. op_ne.json::number[11] `1 != Infinity`: expected Boolean(true), but consteval returned None
  93. op_ne.json::coerce_bool_number[3] `false != NaN`: expected Boolean(true), but consteval returned None
  94. op_ne.json::coerce_string_number[11] `' Infinity ' != Infinity`: expected Boolean(false), but consteval returned None
  95. op_ne.json::coerce_string_number[12] `' -Infinity ' != -Infinity`: expected Boolean(false), but consteval returned None
  96. op_ne.json::coerce_string_number[13] `'NaN' != NaN`: expected Boolean(true), but consteval returned None
  97. op_not.json::not[9] `!!-Infinity`: expected Boolean(true), but consteval returned None
  98. op_not.json::not[10] `!!Infinity`: expected Boolean(true), but consteval returned None
  99. op_not.json::not[11] `!!NaN`: expected Boolean(false), but consteval returned None
 100. operators_case_insensitive.json::lower-vs-upper[0] `'abcdefghijklmnopqrstuvwxyz' == 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'`: expected Boolean(true), got Boolean(false)
 101. operators_case_insensitive.json::lower-vs-upper[1] `'abcdefghijklmnopqrstuvwxyz' != 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'`: expected Boolean(false), got Boolean(true)
 102. operators_case_insensitive.json::lower-vs-upper[3] `'abcdefghijklmnopqrstuvwxyz' <= 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'`: expected Boolean(true), got Boolean(false)
 103. operators_case_insensitive.json::lower-vs-upper[4] `'abcdefghijklmnopqrstuvwxyz' > 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'`: expected Boolean(false), got Boolean(true)
 104. operators_case_insensitive.json::cyrillic-letters[0] `'АБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЬЭЮЯ' == 'абвгдежзийклмнопрстуфхцчшщьэюя'`: expected Boolean(true), got Boolean(false)
 105. operators_case_insensitive.json::upper-vs-lower[0] `'ABCDEFGHIJKLMNOPQRSTUVWXYZ' == 'abcdefghijklmnopqrstuvwxyz'`: expected Boolean(true), got Boolean(false)
 106. operators_case_insensitive.json::upper-vs-lower[1] `'ABCDEFGHIJKLMNOPQRSTUVWXYZ' != 'abcdefghijklmnopqrstuvwxyz'`: expected Boolean(false), got Boolean(true)
 107. operators_case_insensitive.json::upper-vs-lower[2] `'ABCDEFGHIJKLMNOPQRSTUVWXYZ' < 'abcdefghijklmnopqrstuvwxyz'`: expected Boolean(false), got Boolean(true)
 108. operators_case_insensitive.json::upper-vs-lower[5] `'ABCDEFGHIJKLMNOPQRSTUVWXYZ' >= 'abcdefghijklmnopqrstuvwxyz'`: expected Boolean(true), got Boolean(false)
 109. operators_case_insensitive.json::a-z-equivalent-to-A-Z-wrt-chars-between-Z-and-a[4] `'a' < '['`: expected Boolean(true), got Boolean(false)
 110. operators_case_insensitive.json::a-z-equivalent-to-A-Z-wrt-chars-between-Z-and-a[5] `'a' <= '['`: expected Boolean(true), got Boolean(false)
 111. operators_case_insensitive.json::a-z-equivalent-to-A-Z-wrt-chars-between-Z-and-a[6] `'a' > '['`: expected Boolean(false), got Boolean(true)
 112. operators_case_insensitive.json::a-z-equivalent-to-A-Z-wrt-chars-between-Z-and-a[7] `'a' >= '['`: expected Boolean(false), got Boolean(true)
 113. operators_precedence.json::operator_precedence_>_and_<_evaluate_left_to_right[1] `0 > (0 < 1)`: expected result but parse failed:  --> 1:5
  |
1 | 0 > (0 < 1)
  |     ^---
  |
  = expected unary_expr
 114. operators_precedence.json::operator_precedence_>_and_>_evaluate_left_to_right[1] `2 > (2 > 0)`: expected result but parse failed:  --> 1:5
  |
1 | 2 > (2 > 0)
  |     ^---
  |
  = expected unary_expr
 115. operators_precedence.json::operator_precedence_<_and_<=_evaluate_left_to_right[1] `3 < (2 <= 1)`: expected result but parse failed:  --> 1:5
  |
1 | 3 < (2 <= 1)
  |     ^---
  |
  = expected unary_expr
 116. operators_precedence.json::operator_precedence_<_and_<_evaluate_left_to_right[1] `3 < (2 < 1)`: expected result but parse failed:  --> 1:5
  |
1 | 3 < (2 < 1)
  |     ^---
  |
  = expected unary_expr
 117. operators_precedence.json::operator_precedence_<=_and_<_evaluate_left_to_right[1] `3 <= (2 < 1)`: expected result but parse failed:  --> 1:6
  |
1 | 3 <= (2 < 1)
  |      ^---
  |
  = expected unary_expr
 118. operators_precedence.json::operator_precedence_<=_is_higher_than_==[1] `2 <= (3 == true)`: expected result but parse failed:  --> 1:6
  |
1 | 2 <= (3 == true)
  |      ^---
  |
  = expected unary_expr
 119. startsWith.json::basics[9] `startsWith('true', false, true)`: expected parsing error "Too many parameters supplied: " but parsed OK
 120. startsWith.json::basics[10] `startsWith('true')`: expected parsing error "Too few parameters supplied: " but parsed OK
 121. syntax-errors.json::parsing-errors[12] `nonExistentFunction()`: expected parsing error "Unrecognized function: 'nonExistentFunction'. Located at position 1" but parsed OK
 122. syntax-errors.json::parsing-errors[13] `false && nonExistentFunction()`: expected parsing error "Unrecognized function: 'nonExistentFunction'. Located at position 1" but parsed OK
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    test_upstream_kat

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s

error: test failed, to rerun pass `-p github-actions-expressions --test upstream_kat`
vscode ➜ /workspace/.worktrees/add-expressions-tests (dm/expressions-tests) $

So roughly the following categories:

NaN/Infinity not supported in consteval
Case-insensitive string comparison (fixed by Case insensitive comparisons #1687 )
Parenthesized subexpressions not supported
fromJson return value not indexable
Issues with the function arity validation
Unknown functions not rejected
Empty expression rejected
Float formatting divergence
Unknown context not rejected

I will triage the issues, and open the appropriate issues if needed.

woodruffw · 2026-03-04T15:55:02Z

support/sync-expression-tests.py

+def _git_blob_sha(data: bytes) -> str:
+    """Compute the git blob SHA-1 for the given content."""
+    header = f"blob {len(data)}\0".encode()
+    return hashlib.sha1(header + data).hexdigest()


I think this is the wrong level of abstraction -- ideally we'd use the same trick as codeql-injection-sinks here, i.e. just do a sparse checkout of the specific tree we're interested in. That also avoids any GitHub API traffic (since it's git egress), which would count against repo-wide API quotas.

(Specifically, we probably want the equivalent of _clone_actions_codeql, and the PR creation action will do the chore work of skipping the PR if the contents didn't actually change.)

Makes sense!

I liked using the GH API to only update the changed files - but the sparse checkout option is indeed simpler.

Initial introduction of KAT

7d9e9a2

DarkaMaul force-pushed the dm/expressions-tests branch from a16c64b to 7d9e9a2 Compare March 4, 2026 13:45

Fix some errors in the test harness

9c743cf

This was referenced Mar 4, 2026

[BUG]: Expression parser fails on parenthesized compound expressions #1692

Open

[BUG] : NaN and Infinity not recognized as expression literals #1693

Open

woodruffw reviewed Mar 4, 2026

View reviewed changes

DarkaMaul mentioned this pull request Mar 4, 2026

[Meta-BUG]: KAT test failures in the expression evaluator #1694

Open

3 tasks

Use sparse checkout instead

5906604

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Support languageservices expressions test data#1691

Support languageservices expressions test data#1691
DarkaMaul wants to merge 3 commits intozizmorcore:mainfrom
trail-of-forks:dm/expressions-tests

DarkaMaul commented Mar 4, 2026

Uh oh!

DarkaMaul commented Mar 4, 2026

Uh oh!

woodruffw Mar 4, 2026

Uh oh!

DarkaMaul Mar 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

DarkaMaul commented Mar 4, 2026

Pre-submission checks

Summary

Test Plan

Uh oh!

DarkaMaul commented Mar 4, 2026

Uh oh!

woodruffw Mar 4, 2026

Choose a reason for hiding this comment

Uh oh!

DarkaMaul Mar 4, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants