jeudi 31 octobre 2019

Rails tests fail when run as a suite but pass when run individually and also pass on CI

I'm using minitest in Rails 5.2 with Factorybot.

When I run tests individually (eg models/enhanced_object_test.rb) they pass but when I run the whole suite, 'seemingly' unrelated tests start to fail on 'seemingly' unrelated methods.

There are a number of similar questions on SO but they haven't helped. I've tried using DatabaseCleaner between each test but I have the same problem.

I suspect my problem relates to the way I am using factories but I am lost about how to debug.

The error I am getting always relates to the method import_colms_images_to_s3 eg:

error relating to tests/models/info_test_page.rb

test_metadata                                                   FAIL (0.43s)
        not all expectations were satisfied
        unsatisfied expectations:
        - expected exactly once, invoked twice: #<EnhancedObject:0x55652b343e20>.import_colms_images_to_s3(any_parameters)
        - expected exactly once, invoked twice: #<EnhancedObject:0x556527763e28>.import_colms_images_to_s3(any_parameters)
        - expected exactly once, invoked twice: #<EnhancedObject:0x5565289bb170>.import_colms_images_to_s3(any_parameters)

The only place I have an expectation relating to import_colms_images_to_s3 is in 3 tests with the enhanced_object model test eg:

tests/models/enhanced_object_test.rb

## Stubbed the import_colms_images_to_s3 method to avoid the db call to museum_object

test "create enhanced object success" do
    eo = EnhancedObject.new
    eo.name = "test"
    eo.short_description = "shrt dscrptn"
    eo.long_description = "Loooong description"
    eo.museum_unique_id = "O70135"
    eo.colms_images = ["2006ar9715"]
    eo.expects(:import_colms_images_to_s3).returns(true)
    assert eo.save, "Saved enhanced object"
  end

the test for the info_test_page model is:

require 'test_helper'

class InfoPageTest < ActiveSupport::TestCase

  setup do
    @infopage = create(:info_page)
  end

 test "metadata" do
    assert @infopage.metadata.present?
    assert_equal(@infopage.title, @infopage.metadata[:title])
  end
end

This test uses an info_page factory but again this has no connection to enhanced_objects and the failing method.

test/factories/info_pages.rb

FactoryBot.define do
  factory :info_page do
    sequence(:title) { |n| "Info page #{n}" }
    sequence(:slug) { |n| "info-page-#{n}" }
    body { '{"data":[{"type":"heading","data":{"text":"Test info page",'\
         '"format":"html"}},{"type":"text","data":{"text":'\
         '"<p>This is some <b>body</b> text...</p>",'\
         '"format":"html"}},{"type":"quote","data":{"text":'\
         '"<p>This is a quote</p>", "format":"html",'\
         '"cite":"By someone useful."}}]}' }
    background_image {
      fixture_file_upload(
        Rails.root.join('test', 'fixtures', 'header.jpg'),
        'image/jpg') }
    background_image_credit { 'Kermit T. Frog' }
    colour_scheme { ColourScheme::COLOUR_SCHEMES.first }
  end

  trait :without_image do
    background_image { nil }
  end
end

As far as I am concerned, info pages and enhanced objects are unrelated so I can't understand why the tests are interacting in some way.

Final clue is that when the test suite is run on Circle CI the tests pass when run as a suite.

Aucun commentaire:

Enregistrer un commentaire