Conversation
* Ability to specify different compute or service offerings for different types of CKS cluster nodes – worker, master or etcd * Ability to use CKS ready custom templates for CKS cluster nodes --------- Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>
… a kubernetes cluster --------- Co-authored-by: nvazquez <nicovazquez90@gmail.com>
* CKS: Fix ISO attach logic * address comment
…e of external node(s)
|
@blueorangutan package |
|
@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✖️ debian ✔️ suse15. SL-JID 9649 |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #9102 +/- ##
============================================
- Coverage 16.60% 16.57% -0.04%
- Complexity 13924 13968 +44
============================================
Files 5730 5743 +13
Lines 508082 510468 +2386
Branches 61770 62073 +303
============================================
+ Hits 84388 84615 +227
- Misses 414259 416390 +2131
- Partials 9435 9463 +28
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
nvazquez
force-pushed
the
cks-enhancements-upstream
branch
from
5710f92 to
469c08d
Compare
…ter from the same network
--------- Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>
--------- Co-authored-by: nvazquez <nicovazquez90@gmail.com>
…on when External hosts added
|
@blueorangutan package |
|
@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✖️ debian ✔️ suse15. SL-JID 9650 |
bernardodemarco
left a comment
bernardodemarco
left a comment
There was a problem hiding this comment.
Verified that the NPE is fixed and that the scaling in the previous test case works as expected. After the operation
succeeded, this is the current state of the k8s cluster:
(segregated-lab) 🐱 > list kubernetesclusters filter=serviceofferingname,,workerofferingname,controlofferingname,
{
"count": 1,
"kubernetescluster": [
{
"controlofferingname": "control-plane",
"serviceofferingname": "med-offering",
"workerofferingname": "worker-plane"
}
]
}
(segregated-lab) 🐱 > list virtualmachines filter=name,serviceofferingname,
{
"count": 2,
"virtualmachine": [
{
"name": "k8ss-control-1977de3df05",
"serviceofferingname": "med-offering"
},
{
"name": "k8ss-node-1977de40dc8",
"serviceofferingname": "med-offering"
}
]
}IMO, it is a little bit confusing that it is informed for the users that the worker and control offerings are respectively equal to worker-plane and control-plane, but their actual offering is med-offering. The worker_node_service_offering_id and control_node_service_offering_id become unused in this case.
I believe that it would be interesting for us to handle these scenarios. The cluster's metadata should be consistent with the actual computing resources that are allocated for it.
| ALTER TABLE `cloud`.`kubernetes_cluster` ADD CONSTRAINT `fk_cluster__control_node_service_offering_id` FOREIGN KEY `fk_cluster__control_node_service_offering_id`(`control_node_service_offering_id`) REFERENCES `service_offering`(`id`) ON DELETE CASCADE; | ||
| ALTER TABLE `cloud`.`kubernetes_cluster` ADD CONSTRAINT `fk_cluster__worker_node_service_offering_id` FOREIGN KEY `fk_cluster__worker_node_service_offering_id`(`worker_node_service_offering_id`) REFERENCES `service_offering`(`id`) ON DELETE CASCADE; | ||
| ALTER TABLE `cloud`.`kubernetes_cluster` ADD CONSTRAINT `fk_cluster__etcd_node_service_offering_id` FOREIGN KEY `fk_cluster__etcd_node_service_offering_id`(`etcd_node_service_offering_id`) REFERENCES `service_offering`(`id`) ON DELETE CASCADE; | ||
| ALTER TABLE `cloud`.`kubernetes_cluster` ADD CONSTRAINT `fk_cluster__control_node_template_id` FOREIGN KEY `fk_cluster__control_node_template_id`(`control_node_template_id`) REFERENCES `vm_template`(`id`) ON DELETE CASCADE; | ||
| ALTER TABLE `cloud`.`kubernetes_cluster` ADD CONSTRAINT `fk_cluster__worker_node_template_id` FOREIGN KEY `fk_cluster__worker_node_template_id`(`worker_node_template_id`) REFERENCES `vm_template`(`id`) ON DELETE CASCADE; | ||
| ALTER TABLE `cloud`.`kubernetes_cluster` ADD CONSTRAINT `fk_cluster__etcd_node_template_id` FOREIGN KEY `fk_cluster__etcd_node_template_id`(`etcd_node_template_id`) REFERENCES `vm_template`(`id`) ON DELETE CASCADE; |
There was a problem hiding this comment.
Should we have the ON DELETE CASCADE here?
When deleting a service offering, for instance, the kubernetes cluster will be deleted from the DB, right?
There was a problem hiding this comment.
ON DELETE CASCADE has no effect, as ACS never removes the records for templates or server offerings, just marks them as removed.
There was a problem hiding this comment.
Yes, ACS currently soft deletes such records (only mark them as removed in the DB). However, removing the ON DELETE CASCADE could avoid accidental deletion of records by the operator (i.e., accidentally executing a DELETE query in the DB)
|
Thanks @bernardodemarco - in the UI we are preventing that case as the user is requested to set a node per offering instead of a global one, do you think we can add the same check for the API? 
|
@nvazquez, yes
The problem with this approach is that it would break backwards compatibility, right? Prior to the changes, the |
|
@bernardodemarco I think it doesn't break the backwards compatibility as previously created CKS clusters or new clusters not using the advanced options remain the same, the scale works providing the global service offering. However, the use case you present seems to be valid, I suggested adding the check to match the UI in which case users can set the same offering per node type to achieve the same operation, but will explore a solution |
…obal service offering has been provided on scale
|
@bernardodemarco I've addressed the use case:
|
|
@blueorangutan package |
|
@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13822 |
bernardodemarco
left a comment
bernardodemarco
left a comment
There was a problem hiding this comment.
@nvazquez, nice, verified that the cluster's metadata is consistent with the actual offerings applied to each node.
Here are other tests that I performed:
- Created a k8s cluster with a global offering (
min-k8s-offering) - Changed the offerings of the control and worker nodes by manipulating the
nodeofferingsparameter:
(segregated-lab) 🐱 > list virtualmachines filter=name,serviceofferingname,
{
"count": 2,
"virtualmachine": [
{
"name": "k8s-control-1978362ca43",
"serviceofferingname": "control-plane"
},
{
"name": "k8s-node-1978362fcd6",
"serviceofferingname": "worker-plane"
}
]
}
(segregated-lab) 🐱 > list kubernetesclusters filter=name,serviceofferingname,workerofferingname,controlofferingname,
{
"count": 1,
"kubernetescluster": [
{
"controlofferingname": "control-plane",
"name": "k8s",
"serviceofferingname": "min-k8s-offering",
"workerofferingname": "worker-plane"
}
]
}- Changed the global offering of the cluster back to the
min-k8s-offering:
(segregated-lab) 🐱 > scale kubernetescluster id=83b45929-6c50-4002-86b0-83db70a6f2d2 serviceofferingid=1494d756-55b2-4b0e-a062-6f8420e5becf
# (...)
(segregated-lab) 🐱 > list virtualmachines filter=name,serviceofferingname,
{
"count": 2,
"virtualmachine": [
{
"name": "k8s-control-1978362ca43",
"serviceofferingname": "control-plane"
},
{
"name": "k8s-node-1978362fcd6",
"serviceofferingname": "worker-plane"
}
]
}
(segregated-lab) 🐱 > list kubernetesclusters filter=name,serviceofferingname,workerofferingname,controlofferingname,
{
"count": 1,
"kubernetescluster": [
{
"controlofferingname": "control-plane",
"name": "k8s",
"serviceofferingname": "min-k8s-offering",
"workerofferingname": "worker-plane"
}
]
}
- A successful response was returned, but the nodes were not actually scaled. It would be interesting to return an error message to the end user in this scenario.
With the same cluster, scaled its global offering to the med-offering. Verified that the operation was successfully executed:
(segregated-lab) 🐱 > list kubernetesclusters filter=name,serviceofferingname,workerofferingname,controlofferingname,
{
"count": 1,
"kubernetescluster": [
{
"controlofferingname": "med-offering",
"name": "k8s",
"serviceofferingname": "med-offering",
"workerofferingname": "med-offering"
}
]
}
(segregated-lab) 🐱 > list virtualmachines filter=name,serviceofferingname,
{
"count": 2,
"virtualmachine": [
{
"name": "k8s-control-1978362ca43",
"serviceofferingname": "med-offering"
},
{
"name": "k8s-node-1978362fcd6",
"serviceofferingname": "med-offering"
}
]
}Successfully scaled only the control nodes:
(segregated-lab) 🐱 > list kubernetesclusters filter=name,serviceofferingname,workerofferingname,controlofferingname,
{
"count": 1,
"kubernetescluster": [
{
"controlofferingname": "max-offering",
"name": "k8s",
"serviceofferingname": "med-offering",
"workerofferingname": "med-offering"
}
]
}
(segregated-lab) 🐱 > list virtualmachines filter=name,serviceofferingname,
{
"count": 2,
"virtualmachine": [
{
"name": "k8s-control-1978362ca43",
"serviceofferingname": "max-offering"
},
{
"name": "k8s-node-1978362fcd6",
"serviceofferingname": "med-offering"
}
]
}Successfully scaled only the worker nodes:
(segregated-lab) 🐱 > list virtualmachines filter=name,serviceofferingname,
{
"count": 2,
"virtualmachine": [
{
"name": "k8s-control-1978362ca43",
"serviceofferingname": "max-offering"
},
{
"name": "k8s-node-1978362fcd6",
"serviceofferingname": "max-offering"
}
]
}
(segregated-lab) 🐱 > list kubernetesclusters filter=name,serviceofferingname,workerofferingname,controlofferingname,
{
"count": 1,
"kubernetescluster": [
{
"controlofferingname": "max-offering",
"name": "k8s",
"serviceofferingname": "med-offering",
"workerofferingname": "max-offering"
}
]
}Scaled the cluster specifying the global offering and specific offerings for the worker and control nodes. Verified that the global offering was ignored, as expected, and the nodes were scaled to the offerings specified in the nodeofferings parameter.
Deployed a new cluster, with the min-k8s-offering global offering. Scaled the cluster updating the offering of the worker nodes and the cluster's global offering:
scale kubernetescluster id=d43f889e-341e-465d-983f-e5f1537dc991 serviceofferingid=345ec1d7-7c43-4345-9acf-affdebb6ef38 nodeofferings[0].node="worker" nodeofferings[0].offering="9a5a2e85-3d99-468e-9467-fa4aae7a37a6"Verified that the worker offering was correctly scaled. It was expected, however, the offering for the control nodes to be scaled to the control-plane one (345ec1d7-7c43-4345-9acf-affdebb6ef38). However, it was not changed:
(segregated-lab) 🐱 > list virtualmachines filter=name,serviceofferingname,
{
"count": 2,
"virtualmachine": [
{
"name": "k8sss-control-197837dbc1f",
"serviceofferingname": "min-k8s-offering"
},
{
"name": "k8sss-node-197837def68",
"serviceofferingname": "worker-plane"
}
]
}
(segregated-lab) 🐱 > list kubernetesclusters filter=name,serviceofferingname,workerofferingname,controlofferingname,
{
"count": 1,
"kubernetescluster": [
{
"name": "k8sss",
"serviceofferingname": "min-k8s-offering",
"workerofferingname": "worker-plane"
}
]
}Verified a similar scenario of the previous one. Updated the global offering and the control offering at the same time. The control offering was scaled successfully, but the global offering was ignored:
(segregated-lab) 🐱 > scale kubernetescluster id=d43f889e-341e-465d-983f-e5f1537dc991 serviceofferingid=730f4c2b-7729-425f-81d7-fbb556d0eef3 nodeofferings[0].node="control" nodeofferings[0].offering="345ec1d7-7c43-4345-9acf-affdebb6ef38"
# (...)
(segregated-lab) 🐱 > list virtualmachines filter=name,serviceofferingname,
{
"count": 2,
"virtualmachine": [
{
"name": "k8sss-control-197837dbc1f",
"serviceofferingname": "control-plane"
},
{
"name": "k8sss-node-197837def68",
"serviceofferingname": "worker-plane"
}
]
}
(segregated-lab) 🐱 > list kubernetesclusters filter=name,serviceofferingname,workerofferingname,controlofferingname,
{
"count": 1,
"kubernetescluster": [
{
"controlofferingname": "control-plane",
"name": "k8sss",
"serviceofferingname": "min-k8s-offering",
"workerofferingname": "worker-plane"
}
]
}
|
Many thanks @bernardodemarco for your testing. In my understanding your case 3-4 are expected as the global offering was actually the same as the current one, I can add a message on the API response for that case. For the last 2 cases I think that is also expected as per the parameter description:
If you agree we can proceed with this PR and discuss these corner cases on separate issue/PR, as the user would have a workaround by providing the offerings at the same time for each node type |
bernardodemarco
left a comment
bernardodemarco
left a comment
There was a problem hiding this comment.
In my understanding your case 3-4 are expected as the global offering was actually the same as the current one, I can add a message on the API response for that case.
@nvazquez, yes, it would be nice.
For the last 2 cases I think that is also expected as per the parameter description:
Yes, I agree. I hadn’t paid attention to the parameter description, so that shouldn’t be a problem at all.
@nvazquez, @Pearl1594, @sureshanaparti, @weizhouapache, @DaanHoogland, @kiranchavala, as for the use case of specifying segregated compute offerings for each Kubernetes cluster plane, it looks good to me.
Regarding the other eight use cases addressed by this PR, unfortunately, I don’t have time to test them thoroughly enough to cover all possible workflows or to review more than 7,000 lines of code. Thus, I won’t be providing an opinion on those parts.
|
Merging this, based on reviews and tests. |
github-project-automation
bot
moved this from In Progress
to Done
in Apache CloudStack 4.21.0
|
Hi @bernardodemarco I have created the following PR to check the scaling service offerings prior to scaling and in case of providing the same offering then fail with a proper message: #11063. Thanks again for your review! |
@nvazquez, nice, thanks! I'll review and test it ASAP |
10 participants
Description
Design Document: https://cwiki.apache.org/confluence/display/CLOUDSTACK/CKS+Enhancements
Documentation PR: apache/cloudstack-documentation#458
This PR extends the CloudStack Kubernetes Service functionalities, matching these requirements:
Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Bug Severity
Screenshots (if appropriate):
How Has This Been Tested?
In testing on vCenter 7.0 environment + NSX SDN
How did you try to break this feature and the system with this change?