Resource Request: Accelerator quotas for AI Conformance CI jobs

[janetkuo]

AI Conformance is currently designing the automated testing framework for Kubernetes AI Conformance to move away from our current process of manual self-attestation (see our design doc).

As part of this effort, we plan to set up CI jobs (presubmits and periodics) that require access to specialized hardware. Specifically, we need trivial quotas for accelerators (e.g., nvidia-tesla-t4 or L4) to verify basic functionality (e.g., secure accelerator access) without incurring high costs.

I found existing GPU jobs in GCP and AWS. Should our new AI conformance jobs share the existing approach/resources used by those jobs, or provision new ones?

Cost estimate for accelerators:

Variable	Example Value (T4 GPU)
Cost per Hour	~$0.35 (on-demand) or ~$0.11 (preemptible/spot) / instance
Job Frequency	1 periodic + ~0.1 presubmit / day
Duration per Job	~30 min
Monthly Estimate	($0.11/hr * 2 instances * 1.1 jobs/day * 0.5 hours) * 30 days = ~$3.63 / month (note: this assumes using spot instance; duration per job might increase as we add more AI conformance tests)

@ritazh @mfahlandt @terrytangyuan @dims @BenTheElder @ameukam

[ameukam]

cc @kubernetes/sig-k8s-infra-leads

[ameukam]

+1 for me. On GCP, we have a specific GPU-related projects that can be used. On AWS, the boskos AWS accounts can utilized.

Reminder that those resources are on-demand and subject to unavailability. we don't control the lifecycle of those resources.

[hakman]

+1 from me also

[dims]

+1 (non-binding!)

[xmudrii]

+1 from me

[BenTheElder]

+1 (non-binding)

+1 for me. On GCP, we have a specific GPU-related projects that can be used. On AWS, the boskos AWS accounts can utilized.

We should probably make sure the new + existing GPU jobs use a prow queue given the relatively small number of projects versus the generic e2e projects. This is a minor optimization we can do later to avoid failures at the boskos acquire time when things get busy versus letting the jobs queue. Either way we can react by revisting capacity if this occurs.