make test case for SCS-0123 "object-store" no longer mandatory

[depressiveRobot]

Since several providers have complained that the test case for SCS-0123 "object-store" does not meet the standard (being mandatory). This will set it to recommended according to the standard. This PR will make it no longer mandatory.

"object-store" was mandatory before, as the test case for "S3" relies on “object-store” service API. As "S3" is currently skipped if no "object-store" is available, we can do this change to be in line with the standard. However, this should not prevent anyone from offering S3. This problem is already tackled in #1004 and #1093.

Fixes #1110

[mbuechse]

This is quite the substantial change. I know that the tests were wrong (too strict), and I think we should find a way that's the right amount of strict, but the question is should we really disable them until we find the right way? Mind you that object-store is indeed NOT recommended, but only listed as "supported" (from my POV, every service that is neither mandatory nor supported should actually trigger a warning, but that is a topic for another time).

[fkr]

Likely I'm misunderstanding something: if object store is only in the supported list, why do we even test for it (and not just remove it from the test cases list). I understand that moving it to 'recommended' is not ideal, since it is not recommended (hence the question: why not remove it).

The hurdle that the assessment body in the certification has is that the companies running these checks complain to us that we test something that is not mandatory and it throws an error.

[mbuechse]

No, you're not misunderstanding. I think it should be removed. But if the current -- flawed -- test is no longer acceptable to partners, it needs to be replaced with an acceptable variant. This is #1110 -- I did expect this to happen, but at the time, no one had complained. Now people do complain, and now we need to address this. Question is how. Partners should tell us how they think we are supposed to check their S3...

[fkr]

Thanks @mbuechse for confirming.

[berendt]

I think the object-store service is a good indicator that there is a S3 endpoint on an OpenStack environment. I would propose to remove the scs-0123-service-object-store test and to only check for the existence of the object-store service user as pre-check of scs-0123-swift-s3.

[mbuechse]

I'm afraid I don't follow 100%.

What do you mean by service user? Apparently, you're not referring to the catalog entry?

What is to happen when the user isn't found? Currently, scs-0123-swift-s3 will be marked as passed when object-store isn't found, because this testcase only checks whether Swift and S3 show the same data. This testcase is highly controversial though, because the standard doesn't really motivate its existence. See this comment in 1110.

[depressiveRobot]

I think the object-store service is a good indicator that there is a S3 endpoint on an OpenStack environment. I would propose to remove the scs-0123-service-object-store test and to only check for the existence of the object-store service user as pre-check of scs-0123-swift-s3.

As I understand it, this is now the case, see #1063. However, this means that the S3 test is still dependent on the object-store service.

@berendt @fzakfeld @chrisschwa Is there really no other way? How do you make your S3 service discoverable? Please join the discussion in #1004.