Skip to content

Enable test failures on TestRandomizedSchema#369

Open
tigrannajaryan wants to merge 1 commit intomainfrom
tigran/enable-randomized-failure
Open

Enable test failures on TestRandomizedSchema#369
tigrannajaryan wants to merge 1 commit intomainfrom
tigran/enable-randomized-failure

Conversation

@tigrannajaryan
Copy link
Collaborator

This was disabled previously and failures were merely showing as error messages in the output. The codebase is stable enough now that prolonged runs of TestRandomizedSchema no longer fail. We are now enabling failures so that any new random schema failure is caught immediately.

This was disabled previously and failures were merely showing
as error messages in the output. The codebase is stable enough
now that prolonged runs of TestRandomizedSchema no longer fail.
We are now enabling failures so that any new random schema
failure is caught immediately.
@github-actions
Copy link

Benchmark Result

Benchmark diff with base branch
goos: linux
goarch: amd64
pkg: github.com/splunk/stef/benchmarks
cpu: AMD EPYC 7763 64-Core Processor                
                                                 │ bench-main.txt │           bench-new.txt            │
                                                 │     sec/op     │    sec/op     vs base              │
SerializeNative/STEF/serialize-4                     7.685m ±  4%   8.301m ± 24%  +8.02% (p=0.041 n=6)
SerializeNative/STEFU/serialize-4                    34.33m ±  2%   34.74m ±  1%       ~ (p=0.093 n=6)
DeserializeNative/STEF/deser-4                       2.423m ±  2%   2.425m ±  1%       ~ (p=0.937 n=6)
DeserializeNative/STEFU/deser-4                      7.246m ±  1%   7.253m ±  0%       ~ (p=0.699 n=6)
SerializeFromPdata/STEF/serialize-4                  140.3m ±  5%   138.6m ±  4%       ~ (p=0.180 n=6)
SerializeFromPdata/STEFU/serialize-4                 34.21m ±  1%   34.23m ±  1%       ~ (p=0.394 n=6)
DeserializeToPdata/STEF/deserialize-4                45.07m ±  1%   45.36m ±  1%       ~ (p=0.394 n=6)
DeserializeToPdata/STEFU/deserialize-4               62.81m ±  1%   62.68m ±  1%       ~ (p=0.394 n=6)
STEFReaderRead-4                                     2.487m ±  2%   2.506m ±  1%       ~ (p=0.310 n=6)
STEFSerializeMultipart/astronomy-otelmetrics-4        3.426 ± 19%    3.314 ± 21%       ~ (p=0.132 n=6)
STEFDeserializeMultipart/astronomy-otelmetrics-4     76.92m ± 10%   72.65m ± 11%       ~ (p=0.394 n=6)
ReadSTEF-4                                           2.586m ±  2%   2.583m ±  1%       ~ (p=0.937 n=6)
ReadSTEFZ-4                                          3.370m ±  1%   3.278m ±  1%  -2.72% (p=0.002 n=6)
ReadSTEFZWriteSTEF-4                                 7.966m ±  2%   7.781m ±  1%  -2.33% (p=0.002 n=6)
geomean                                              20.81m         20.74m        -0.37%

                                                 │ bench-main.txt │           bench-new.txt            │
                                                 │   sec/point    │  sec/point    vs base              │
SerializeNative/STEF/serialize-4                     114.9n ±  4%   124.2n ± 24%  +8.05% (p=0.041 n=6)
SerializeNative/STEFU/serialize-4                    513.5n ±  2%   519.7n ±  1%       ~ (p=0.093 n=6)
DeserializeNative/STEF/deser-4                       36.24n ±  2%   36.27n ±  1%       ~ (p=0.937 n=6)
DeserializeNative/STEFU/deser-4                      108.4n ±  1%   108.5n ±  0%       ~ (p=0.732 n=6)
SerializeFromPdata/STEF/serialize-4                  2.101µ ±  5%   2.074µ ±  4%       ~ (p=0.197 n=6)
SerializeFromPdata/STEFU/serialize-4                 512.0n ±  1%   512.5n ±  1%       ~ (p=0.420 n=6)
DeserializeToPdata/STEF/deserialize-4                674.5n ±  1%   679.0n ±  1%       ~ (p=0.396 n=6)
DeserializeToPdata/STEFU/deserialize-4               940.0n ±  1%   938.0n ±  1%       ~ (p=0.411 n=6)
STEFReaderRead-4                                     37.19n ±  1%   37.49n ±  1%       ~ (p=0.310 n=6)
STEFSerializeMultipart/astronomy-otelmetrics-4       4.355µ ± 19%   4.212µ ± 21%       ~ (p=0.132 n=6)
STEFDeserializeMultipart/astronomy-otelmetrics-4     97.77n ± 10%   92.34n ± 11%       ~ (p=0.394 n=6)
ReadSTEF-4                                           38.71n ±  2%   38.65n ±  1%       ~ (p=0.974 n=6)
ReadSTEFZ-4                                          50.44n ±  1%   49.06n ±  1%  -2.73% (p=0.002 n=6)
ReadSTEFZWriteSTEF-4                                 119.2n ±  2%   116.5n ±  1%  -2.31% (p=0.002 n=6)
geomean                                              219.0n         218.2n        -0.37%

                                                 │ bench-main.txt │            bench-new.txt             │
                                                 │      B/op      │     B/op      vs base                │
SerializeNative/STEF/serialize-4                     3.337Mi ± 0%   3.338Mi ± 0%       ~ (p=0.589 n=6)
SerializeNative/STEFU/serialize-4                    7.557Mi ± 0%   7.557Mi ± 0%       ~ (p=0.184 n=6)
DeserializeNative/STEF/deser-4                       951.4Ki ± 0%   951.4Ki ± 0%       ~ (p=1.000 n=6) ¹
DeserializeNative/STEFU/deser-4                      1.715Mi ± 0%   1.715Mi ± 0%       ~ (p=0.242 n=6)
SerializeFromPdata/STEF/serialize-4                  76.55Mi ± 0%   76.55Mi ± 0%       ~ (p=0.524 n=6)
SerializeFromPdata/STEFU/serialize-4                 7.557Mi ± 0%   7.557Mi ± 0%       ~ (p=0.558 n=6)
DeserializeToPdata/STEF/deserialize-4                31.99Mi ± 0%   31.99Mi ± 0%       ~ (p=0.299 n=6)
DeserializeToPdata/STEFU/deserialize-4               38.88Mi ± 0%   38.88Mi ± 0%       ~ (p=0.764 n=6)
STEFReaderRead-4                                     953.1Ki ± 0%   953.1Ki ± 0%       ~ (p=1.000 n=6) ¹
STEFSerializeMultipart/astronomy-otelmetrics-4       3.384Gi ± 0%   3.383Gi ± 0%       ~ (p=0.310 n=6)
STEFDeserializeMultipart/astronomy-otelmetrics-4     20.30Mi ± 0%   20.30Mi ± 0%       ~ (p=0.429 n=6)
ReadSTEF-4                                           953.2Ki ± 0%   953.2Ki ± 0%       ~ (p=0.890 n=6)
ReadSTEFZ-4                                          10.29Mi ± 0%   10.29Mi ± 0%       ~ (p=0.937 n=6)
ReadSTEFZWriteSTEF-4                                 13.44Mi ± 0%   13.44Mi ± 0%       ~ (p=0.504 n=6)
geomean                                              10.66Mi        10.66Mi       +0.00%
¹ all samples are equal

                                                 │ bench-main.txt │            bench-new.txt            │
                                                 │   allocs/op    │  allocs/op   vs base                │
SerializeNative/STEF/serialize-4                      2.651k ± 0%   2.651k ± 0%       ~ (p=0.615 n=6)
SerializeNative/STEFU/serialize-4                      884.5 ± 0%    884.0 ± 0%       ~ (p=1.000 n=6)
DeserializeNative/STEF/deser-4                         463.0 ± 0%    463.0 ± 0%       ~ (p=1.000 n=6) ¹
DeserializeNative/STEFU/deser-4                        496.0 ± 0%    496.0 ± 0%       ~ (p=1.000 n=6) ¹
SerializeFromPdata/STEF/serialize-4                   134.7k ± 0%   134.7k ± 0%       ~ (p=1.000 n=6)
SerializeFromPdata/STEFU/serialize-4                   886.0 ± 0%    886.0 ± 0%       ~ (p=1.000 n=6) ¹
DeserializeToPdata/STEF/deserialize-4                 756.2k ± 0%   756.2k ± 0%       ~ (p=1.000 n=6)
DeserializeToPdata/STEFU/deserialize-4                944.9k ± 0%   944.9k ± 0%       ~ (p=1.000 n=6) ¹
STEFReaderRead-4                                       463.0 ± 0%    463.0 ± 0%       ~ (p=1.000 n=6) ¹
STEFSerializeMultipart/astronomy-otelmetrics-4        13.15M ± 0%   13.15M ± 0%       ~ (p=0.058 n=6)
STEFDeserializeMultipart/astronomy-otelmetrics-4      1.956k ± 0%   1.956k ± 0%       ~ (p=1.000 n=6)
ReadSTEF-4                                             464.0 ± 0%    464.0 ± 0%       ~ (p=1.000 n=6) ¹
ReadSTEFZ-4                                            502.0 ± 0%    502.0 ± 0%       ~ (p=1.000 n=6) ¹
ReadSTEFZWriteSTEF-4                                  1.231k ± 0%   1.231k ± 0%       ~ (p=1.000 n=6) ¹
geomean                                               6.305k        6.305k       -0.00%
¹ all samples are equal
Benchmark result
benchstat bench-new.txt
goos: linux
goarch: amd64
pkg: github.com/splunk/stef/benchmarks
cpu: AMD EPYC 7763 64-Core Processor                
                                                 │ bench-new.txt │
                                                 │    sec/op     │
SerializeNative/STEF/serialize-4                    8.301m ± 24%
SerializeNative/STEFU/serialize-4                   34.74m ±  1%
DeserializeNative/STEF/deser-4                      2.425m ±  1%
DeserializeNative/STEFU/deser-4                     7.253m ±  0%
SerializeFromPdata/STEF/serialize-4                 138.6m ±  4%
SerializeFromPdata/STEFU/serialize-4                34.23m ±  1%
DeserializeToPdata/STEF/deserialize-4               45.36m ±  1%
DeserializeToPdata/STEFU/deserialize-4              62.68m ±  1%
STEFReaderRead-4                                    2.506m ±  1%
STEFSerializeMultipart/astronomy-otelmetrics-4       3.314 ± 21%
STEFDeserializeMultipart/astronomy-otelmetrics-4    72.65m ± 11%
ReadSTEF-4                                          2.583m ±  1%
ReadSTEFZ-4                                         3.278m ±  1%
ReadSTEFZWriteSTEF-4                                7.781m ±  1%
geomean                                             20.74m

                                                 │ bench-new.txt │
                                                 │   sec/point   │
SerializeNative/STEF/serialize-4                    124.2n ± 24%
SerializeNative/STEFU/serialize-4                   519.7n ±  1%
DeserializeNative/STEF/deser-4                      36.27n ±  1%
DeserializeNative/STEFU/deser-4                     108.5n ±  0%
SerializeFromPdata/STEF/serialize-4                 2.074µ ±  4%
SerializeFromPdata/STEFU/serialize-4                512.5n ±  1%
DeserializeToPdata/STEF/deserialize-4               679.0n ±  1%
DeserializeToPdata/STEFU/deserialize-4              938.0n ±  1%
STEFReaderRead-4                                    37.49n ±  1%
STEFSerializeMultipart/astronomy-otelmetrics-4      4.212µ ± 21%
STEFDeserializeMultipart/astronomy-otelmetrics-4    92.34n ± 11%
ReadSTEF-4                                          38.65n ±  1%
ReadSTEFZ-4                                         49.06n ±  1%
ReadSTEFZWriteSTEF-4                                116.5n ±  1%
geomean                                             218.2n

                                                 │ bench-new.txt │
                                                 │     B/op      │
SerializeNative/STEF/serialize-4                    3.338Mi ± 0%
SerializeNative/STEFU/serialize-4                   7.557Mi ± 0%
DeserializeNative/STEF/deser-4                      951.4Ki ± 0%
DeserializeNative/STEFU/deser-4                     1.715Mi ± 0%
SerializeFromPdata/STEF/serialize-4                 76.55Mi ± 0%
SerializeFromPdata/STEFU/serialize-4                7.557Mi ± 0%
DeserializeToPdata/STEF/deserialize-4               31.99Mi ± 0%
DeserializeToPdata/STEFU/deserialize-4              38.88Mi ± 0%
STEFReaderRead-4                                    953.1Ki ± 0%
STEFSerializeMultipart/astronomy-otelmetrics-4      3.383Gi ± 0%
STEFDeserializeMultipart/astronomy-otelmetrics-4    20.30Mi ± 0%
ReadSTEF-4                                          953.2Ki ± 0%
ReadSTEFZ-4                                         10.29Mi ± 0%
ReadSTEFZWriteSTEF-4                                13.44Mi ± 0%
geomean                                             10.66Mi

                                                 │ bench-new.txt │
                                                 │   allocs/op   │
SerializeNative/STEF/serialize-4                     2.651k ± 0%
SerializeNative/STEFU/serialize-4                     884.0 ± 0%
DeserializeNative/STEF/deser-4                        463.0 ± 0%
DeserializeNative/STEFU/deser-4                       496.0 ± 0%
SerializeFromPdata/STEF/serialize-4                  134.7k ± 0%
SerializeFromPdata/STEFU/serialize-4                  886.0 ± 0%
DeserializeToPdata/STEF/deserialize-4                756.2k ± 0%
DeserializeToPdata/STEFU/deserialize-4               944.9k ± 0%
STEFReaderRead-4                                      463.0 ± 0%
STEFSerializeMultipart/astronomy-otelmetrics-4       13.15M ± 0%
STEFDeserializeMultipart/astronomy-otelmetrics-4     1.956k ± 0%
ReadSTEF-4                                            464.0 ± 0%
ReadSTEFZ-4                                           502.0 ± 0%
ReadSTEFZWriteSTEF-4                                 1.231k ± 0%
geomean                                              6.305k

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant