While writing tests using Ginkgo framework I noticed, that pressing C-c to terminate a running suite generates false positive.
When you look at the code you will notice this test should fail after 5 seconds. When I terminate it after 2 seconds, the suite fails, but in the results, there is 1 passed test and 0 failed.
Same behavior for go versions 1.11.4 and 1.12.4 on Debian Stretch and Ubuntu 18.04.
Suite code (autogenerated using ginkgo bootstrap
):
package hmmm_test
import (
"testing"
. "github.com/onsi/ginkgo"
. "github.com/onsi/gomega"
)
func TestHmmm(t *testing.T) {
RegisterFailHandler(Fail)
RunSpecs(t, "Hmmm Suite")
}
Test code:
package hmmm_test
import (
"time"
. "github.com/onsi/ginkgo"
. "github.com/onsi/gomega"
)
var _ = Describe("Hmmm", func() {
Context("Dummy test", func() {
It("should fail after 5 seconds", func() {
time.Sleep(5 * time.Second)
Expect(1).NotTo(Equal(1))
})
})
})
Output when the test runs for 5 seconds (correct one):
$ ginkgo
Running Suite: Hmmm Suite
=========================
Random Seed: 1555580607
Will run 1 of 1 specs
• Failure [5.001 seconds]
Hmmm
/tmp/hmmm/hmmm_test.go:10
Dummy test
/tmp/hmmm/hmmm_test.go:11
should fail after 5 seconds [It]
/tmp/hmmm/hmmm_test.go:12
Expected
<int>: 1
not to equal
<int>: 1
/tmp/hmmm/hmmm_test.go:14
------------------------------
Summarizing 1 Failure:
[Fail] Hmmm Dummy test [It] should fail after 5 seconds
/tmp/hmmm/hmmm_test.go:14
Ran 1 of 1 Specs in 5.002 seconds
FAIL! -- 0 Passed | 1 Failed | 0 Pending | 0 Skipped
--- FAIL: TestHmmm (5.00s)
FAIL
Ginkgo ran 1 suite in 5.665592703s
Test Suite Failed
Output when the test is terminated before it finishes (false positive):
$ ginkgo
Running Suite: Hmmm Suite
=========================
Random Seed: 1555580763
Will run 1 of 1 specs
^C
Ran 1 of 1 Specs in 1.187 seconds
FAIL! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped
Ginkgo ran 1 suite in 1.85541211s
Test Suite Failed
I expect the output to be something like: FAIL! -- 0 Passed | 1 Failed | 0 Pending | 0 Skipped
Or 1 skipped or pending, but not Passed
especially that the test is written to fail.
Actual output suggests failure, but also that all tests... passed: FAIL! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped
Am I missing something?
Aucun commentaire:
Enregistrer un commentaire