Compile farm for Hosts
Preamble
A Host is an organization or an individual that provides resources for the compile farm. Such resources typically include:
- hosting space, electrical power, and cooling in a datacenter or a similar facility (server room, cabinet...)
- network connectivity
- "remote hands" service, e.g. the ability to power cycle a crashed machine, change a failed disk...
- possibly some hardware (servers, boards)
Simply donating hardware (without hosting it) is welcome, but is not enough to be considered a Host.
What the compile farm expects from a Host
Price
The resources must be made available to the compile farm free of charge. This is considered an in-kind donation to support the operation of the farm.
Availability
The compile farm has no hard availability requirements. As such, it is acceptable for a Host to provide the resources on a best-effort basis.
However, the Host should make reasonable attempts at providing a stable service:
- the Host and its representative must have appropriate authorization to provide the resources.
- when setting up a machine, is it expected to stay available for at least a few years (barring unforeseen circumstances and with a best-effort policy)
- maintenance operations that can have a significant impact on the service should be communicated to the Admin team
- if a machine or the hosting space can no longer be provided to the Compile Farm, it must be communicated as soon as possible to the Admin team
- we may try to find another Host for this machine
In other words, hosting a machine for just a few months is not efficient and should be avoided.
If a Host cannot commit to this level of availability, it is possible to provide resources under an "experimental" status and re-evaluate this status after some time.
Network connectivity
The Host must provide a way to access the hosted machines over SSH from the Internet.
If possible, inbound SSH access should be provided on TCP ports 22 and 443 on each machine. Otherwise, providing SSH on higher port numbers through port redirects is an acceptable workaround.
The Compile Farm Admin team ensures that login is only possible with a SSH key, not with a password, which makes bruteforce SSH attacks imposible.
Outbound connectivity to the Internet should be provided without filtering, because Users need to access a wide variety of online resources through a wide variety of protocols.
If necessary to comply with its security policy, the Host may choose to selectively filter some outbound protocols (such as SMTP or ICMP). The Host must inform the Admin team of such filtering.
Bandwidth limitations need to be communicated before/during onboarding. This means link speed and data transfer limits. Throttling for enforcement is OK, but unavailability due to overages should be avoided.
It is the responsibility of the Host to ensure that the Compile Farm machines are isolated from its own internal services. Example techniques to ensure this include DMZ, VLAN, firewall rules.
Physical and out-of-band access
The Host should provide a contact point so that Admins can request "remote hands" operations on the hosted machines. There are no delay constraints to handle such requests: it is perfectly acceptable to perform "remote hands" operations within a few days or even a few weeks. Ideally, an estimated turnaround time should be provided as part of onboarding.
If the Host cannot perform "remote hands" operations itself, it should allow physical access to one of the Admins. Out-of-band management interfaces (BMC, iLOM, etc.) should be made available via VPN if available.
Remote access for admins
The Admin team requires remote root access over SSH during the whole lifetime of the hosted machines.
Software installation and maintenance
The Host must setup the initial Operating System on the machines, except possibly for third-party machines.
The responsibility of the Host includes:
- installing an Operating System with appropriate support for the hardware
- setting up network connectivity
- configuring the firewall so that the Host's security needs are met, while also permitting the needs of the Compile Farm
- ensuring that the Admin team can access the machine remotely over SSH as root (i.e. allowing the SSH public key of the Admin team)
- make any necessary changes to hardware or software as requested by the Admin team
After the initial installation, during the normal operation of the machine, the Host can perform hardware or firmware upgrades it deems necessary after notifying the Admin team. Trivial updates and reboots do not require notification, but please use sound judgment.
However, the Host should not upgrade to a new major Operating System version. The Host should first coordinate with the Admin team. This aims to ensure a good coverage of OS and OS versions provided in the farm.
Hosting third-party machines
The Host can optionally provide hosting space for third-party machines, i.e. machines donated by other organizations.
If the host is willing to host third-party machines, it should provide the following details to the Admin team:
- the exact location of the hosting facility
- how many machines can be hosted
- which kind of machines can be hosted (rackable, free-form boards, boards in enclosure...)
- what amount of electrical power is available
- whether network configuration needs to be statically defined in advance (MAC addresses, DHCP, ...)
- any other constraints
- a physical address to which to ship machines
What a Host can expect from the Compile Farm
The Host will be clearly credited on the Compile Farm website, unless the Host wishes not to appear publicly.
The Admin team will manage user accounts on the hosted machines.
The Admin team will manage software packages on the hosted machines, except for very exotic systems.
When the Admin team detects an issue with a machine, it will contact the Host through its support channel.
What a Host can expect from farm users
All farm users obtain SSH access to farm machines. They will use this SSH access to download, build, debug software, and possibly interact with external systems (Buildbot...). All network traffic generated by this activity will flow through the Host network.
If the Host has opted in to allow SSH port forwarding, users may also forward traffic from their devices through the Host network.
Users of the compile farm are expected to only use the resources for free software development.
In case of abuse, the Host is encouraged to report the issue to the Admin team, which will take appropriate action: warn the user, and then possibly suspend their account.
Hardware and operating system specifications
The Compile Farm welcomes all hardware, especially exotic hardware. The only hard constraints are:
- the hardware must support a multi-user Unix-like operating system with SSH access.
- the Operating System must be able to run Python 3 (our management system uses Ansible)
Current hardware and operating systems are visible on
https://cfarm.tetaneutral.net/machines/list/ . The farm is open to integrate new hardware and operating systems and the Admin team can work to support them.
In addition, based on operational experience, the following additional guidelines are recommended to make the systems useful and to lower the maintenance load:
- RAM: recommended minimum amount is 1 GB per CPU core (without taking into account hyperthreading/SMT). If several hardware options are possible, a larger amount of RAM is better.
- Mount points: /home should be a separate mount point (partition or dedicated disk)
- Storage for root: minimum 8 GB for embedded devices; minimum 30 GB for larger servers. 40 GB is recommended
- Storage for home: at least 100 GB of local storage per CPU core, more is welcome. Mechanical disks (HDD) or SSD are possible. There is no RAID requirements. If local storage is not available, /home can be mounted over NFS from a server in the same network.
These guidelines are flexible, discussion is welcome with the Admin team for specific cases.
Process for new hosts
Send an email to cfarm-admins at lists.tetaneutral.net with:
- the name of the organization
- a list of resources that can be provided to the farm
- whether SSH port forwarding should be allowed or denied to farm users on your network
- a contact point for administrative questions
- a contact point for support / remote hands with estimated turnaround time