Sysadmin France
***************

This document is for system administrators, webmasters and users of GNU and FSF machines in France.

Introduction
************

No information on the philosophical and political goals of the FSF Europe in general and the FSFE France in particular can be found in this document.

The machines donated to the FSF Europe and hosted in France are part of the set of machines available to the GNU project. As such they follow the same rules and the same policy.

Reference documents describing the GNU environment are http://savannah.gnu.org/projects/sysadmin for system administration information and the Savannah admin guide for usage of the project hosting facility.

The essential URLs related to this document are:

  * Nevrax mrtg

  * Free mrtg

  * Netsaint

  * This document

Hardware List
*************

This list accurately describes the purpose of each piece of harware dedicated to the GNU project in France. It must not describe the interconnection between them. If something is running on these machines that do not match the intented purpose described here, it can be uninstalled without notice.

  * UPS x 3

     Contact : Loic Dachary, Fabien Piuzzi

  * fr.fsf.org

     Web hosting (www.fsfeurope.org, france.fsfeurope.org); Secondary DNS for GNU (ns2.gnu.org); GNU Friends hosting.

          PIII500, 512Mb RAM, 5x9Gb SCSI

     Contact : Loic Dachary, Fabien Piuzzi

  * snail.gnu.org

     Savannah mirror; Audio/Video gatekeeper (gnomemeeting relay);

          Dual PIII 550
          1 GB RAM
          Asus P2B-D
          2 serial ports
          1 parallel port
          2 ethernet 3c905C-TX
          2 20 GB IDE disks
          1 SCSI Adaptec 7892A (rev 02)
          Double power

     Contact : Cyril Bouthors, Rodolphe Quiédeville, Jean-Louis Bergamo, Loic Dachary

  * frog.gnu.org

     Serial console server.

          PII 350
          256 MB RAM
          Asus P2B-D
          2 serial ports
          1 parallel port
          1 ethernet 3c905C-TX
          2 20 GB IDE disks
          Double power

     Contact : Cyril Bouthors, Rodolphe Quiédeville, Jean-Louis Bergamo, Loic Dachary

  * yoda.gnu.org (yoda.ipsyn.net)

     Serial console server.

          Bi PIII 750
          512 MB RAM
          2 serial ports
          1 parallel port
          1 ethernet eepro100
          2 18 GB SCSI disks
          Adaptec aic7896/97 Ultra2 SCSI adapter

     Contact : Jean-Louis Bergamo, Loic Dachary

  * Cisco Catalyst 3500 XL

     Typical session:

          switch> enable
          Password:
          switch# show configuration (show kind of a diff with the defaults)
          switch# configure terminal
          switch(config)# exit
          switch# write memory
          switch# exit

     It is only possible to get access to the switch from the serial line.

     The SNMP server was configured and is available with the GnuRms community.

     Documents : Product description, Product manual

     Contact : Jean-Louis Bergamo, Loic Dachary

  * BlackBox Pow-R-Boot 5 +

     Documents : Product manual

     Contact : Loic Dachary

  * Blackbox ServSwitch

     5 ports.

     Documents : Product manual

     Contact : Loic Dachary

  * Desktop

     Not available on the Internet. Dedicated to office needs of FSFE France.

          Dual PIII 500
          512Mb RAM
          4G SCSI
          8G IDE
          CD PLEXTOR 32X
          CD TEAC CD-R55S
          ATI Mach64
          Voodoo 2
          AWE 64 gold
          IIyama 17"

     Contact : Loic Dachary, Rodolphe Quiédeville, Laurent Guerby

  * Laptop

     Not available on the Internet. Dedicated to Frederic Couchet for permanent activism.

          Inspiron 3500
          128Mb RAM
          6G Disk

     Contact : Loic Dachary, Frederic Couchet

Hardware Setup
**************

Each hardware is mentionned by its name in the see Hardware List. Each location is mentionned by its name in the see Hosting facilities.  The sole purpose of this chapter is to describe the precise location of the corresponding hardware and its physical connections with other hardware.

  * snail.gnu.org is hosted at Free.

          ttyS0 = snail.gnu.org:/dev/ttyS0 -> Cisco Catalyst 3500 XL:console : 9600 8N1,
          	cisco specific cable (flat blue rj45 + db9 adaptator ref 74-0495-01 written on it)
          ttyS1 = frog.gnu.org:/dev/ttyS1 -> snail.gnu.org:/dev/ttyS1 (getty) : 9600 8N1, null modem db9
          power cable is controled by BlackBox Pow-R-Boot 5 +
          ethernet cable is connected to Cisco Catalyst 3500 XL

  * frog.gnu.org is hosted at Free.

          ttyS0 = frog.gnu.org:/dev/ttyS0 -> BlackBox Pow-R-Boot 5 +:console : 9600 8N1, null modem db9
          ttyS1 = frog.gnu.org:/dev/ttyS1 -> snail.gnu.org:/dev/ttyS1 (getty) : 9600 8N1, null modem db9
          power cable is controled by BlackBox Pow-R-Boot 5 +
          ethernet cable is connected to Cisco Catalyst 3500 XL

  * yado.gnu.org(yoda.ipsyn.net) is hosted at Free.

          power cable is controled by BlackBox Pow-R-Boot 5 + (port 3)
          ethernet cable is connected to Cisco Catalyst 3500 XL (port 5)

  * Cisco Catalyst 3500 XL is hosted at Free.

          snail.gnu.org:/dev/ttyS0 -> Cisco Catalyst 3500 XL:console : 9600 8N1,
          	cisco specific cable (flat blue rj45 + db9 adaptator ref 74-0495-01 written on it)

  * BlackBox Pow-R-Boot 5 + is hosted at Free.

          frog.gnu.org:/dev/ttyS0 -> powerboot:console : 9600 8N1, null modem db9

  * Blackbox ServSwitch is hosted at Free.

  * fr.fsf.org is hosted at Nevrax.

          fr.fsf.org power cable is connected to a UPS

  * UPS x 3 are hosted at Nevrax.

  * Desktop is hosted at Loïc Dachary's home.

  * Laptop is travelling with Frederic Couchet.

Hosting facilities
******************

Each section in this chapter describe a location where GNU machines are hosted in France.

If the people and contacts listed here are for some reason unable to fix an urgent problem, one can look into the sysadmin.texi document (entry Nevrax (http://www.nevrax.com/), Free.fr (http://www.free.fr/)) in the private project http://savannah.gnu.org/projects/sysadmin/. This document list additional information that cannot be published to preserve the privacy of the people.

Free
====

Free is one of the largest Internet provider in France. There is little chance that we are able to spot a connectivity problem which they overlook.

The range of IP addresses that is granted to us is:

213.228.62.2 - 213.228.62.14

The reverse is managed by Free and Antoine Levavasseur is the one to ask for a reverse change. The current setup is

     213.228.62.2 snail.gnu.org
     213.228.62.3 snail-ssh.gnu.org
     213.228.62.4 frog.gnu.org
     213.228.62.7 yoda.gnu.org(yoda.ipsyn.net)

     Linx
     124 bd de Verdun
     92400 Courbevoie
     +33 8 04 55 44 11

  * Loic Dachary Work: +33 1 41 66 47 22, Home: +33 1 42 45 07 97. Installed the machines.

  * Cyril Bouthors. Work: +33 1 41 66 47 06. Installed the machines.

  * Antoine Levavasseur. Work: +33 1 49 04 48 81. Works for Free, is our primary contact and arranged for the rack space and bandwidth. He accepts to be disturbed if an operation requires physical access to the machine, as long as it happens very rarely.

Nevrax
======

     Nevrax
     104 rue du Fg St Antoine
     75011 Paris
     +33 1 44 74 83 85

The leased line that connects Nevrax to the Internet is provided by

     COLT Telecommunication
     60 rue de Wattignies - Bat. B
     75012 PARIS
     Email : opi@fr.colt.net and support@fr.colt.net
     Phone : +33 1 44 29 58 99
     Fax   : +33 1 44 29 57 97

Here is a complete list of the mail threads related to solving problems when something goes wrong with the Internet connection.

  * July 14

Here is a list of people who know some about the machine and its connectivity and who actually did something in the past to improve or fix it.

     Fabien Piuzzi Work: +33 1 44 74 83 85 System administrator who is physically near the machine most of the time.

  * Loic Dachary Work: +33 1 41 66 47 22, Home: +33 1 42 45 07 97. Asked Nevrax to host the machine. Installed the machine. Asked Colt to change the reverse for fr.fsf.org.

  * Cyril Bouthors. Work: +33 1 41 66 47 06. Monitors the machine. Knows Colt pretty well. Opened trouble tickets when the line was down.

  * Rodolphe Quiédeville. Work: +33 6 13 79 63 41. Installed the machine. Installed and maintains monitoring tools on fr.fsf.org.

  * Jean-Louis Bergamo. Work: +33 1 53 01 72 13. Installed the machine yoda.gnu.org (yoda.ipsyn.net). Administration of snail,frog and yoda.gnu.org.

  * Joel N. Weber II. Monitors the machine. Opened trouble tickets when the line was down. Calculated the reliability of the Colt connection.

Machine Accounts
****************

System Passwords
================

If you need the password of system users such as root or www on any GNU machines located in France you should ask to the following people:

  * Rodolphe Quiédeville

  * Loic Dachary

  * Jean-Louis Bergamo

  * Cyril Bouthors

Account Create
==============

The user accounts on fr.fsf.org and snail.gnu.org are managed with Savannah. If someone need a shell account on fr.fsf.org or snail.gnu.org she has to get an account on Savannah first. Then she should send a mail to one of the project administrators of the fsffr project. The project administrator adds her as a member of the fsffr project. Within 24 hours a cron job on fr.fsf.org and snail.gnu.org will fetch the new user from Savannah and create the corresponding account.

Account Access
==============

Once the account is created, access to the machines is available using ssh and a public key. The Savannah password will not work on fr.fsf.org or snail.gnu.org. When logged on Savannah it is possible to register one or more public key at Edit SSH Keys.

Account Update
==============

Once a day the ssh public keys of every account on fr.fsf.org or snail.gnu.org are updated from the information fetched on Savannah. If the authorized_keys file of a user is manually updated, it will be overwritten.

Account Delete
==============

When a user is not listed anymore in the fsffr, its account is disabled on fr.fsf.org and snail.gnu.org. The home directory is not deleted and if the user is added again at a later time, she will retrieve his former home directory.

Account www on fr.fsf.org
=========================

Every account created automatically as described above is also granted an access to the www user on fr.fsf..org (and not on snail.gnu.org). www owns the directory in which all the document roots of the apache server are : /home/www.

This is simply done by appending the public keys of all the users to the authorized_keys file of the www user.

Cron Job
========

The /usr/local/bin/savannahusers script does the user account updates. It is run from the /etc/cron.d/savannahusers cron file and spits log information in /var/log/savannahusers.log. The log file is rotated according to the /etc/logrotate.d/savannahusers specification. This holds for both fr.fsf.org and snail.gnu.org.

Savannah does not provide account information to non identified machines. The fr.fsf.org and snail.gnu.org machines are explicitly allowed to retrieve the relevant information. For more information check the Account Management chapter of the Savannah documentation.

The savannahusers script sources can be found in the www project source tree. It was checked out in /usr/local/src/www directory (on fr.fsf.org and snail.gnu.org), together with other GNU specific maintainance scripts.

Security
********

Security alert
==============

When there is a suspicion that a machine was compromised a mail should be sent to FSF France private mailing list and the following people can be contacted.

  * Loic Dachary. All machines.

  * Jean Louis Bergamo. All machines.

  * Rodolphe Quiédeville. All machines.

  * Igor Genibel. fr.fsf.org only.

  * Fabien Piuzzi. fr.fsf.org only (emergency requiring physical access).

  * Antoine Levavasseur. snail.gnu.org frog.gnu.org and yoda.gnu.org only (emergency requiring physical access).

The intrusion related mails are kept in a private mail archive for future reference and are listed here.

  * June 29 (fr.fsf.org).

Firewall
========

Firewall (fr.fsf.org)
---------------------

A shell script applies filtering rules, it's located in fr.fsf.org:/etc/init.d/firewall. This script is compatible with kernel 2.2 and 2.4 and detects the kernel version using uname. ipchains is used with 2.2 and iptables with 2.4.

Warning : only the ipchains (2.2) section is well configured.

The policy applied is to close all TCP/UDP port by default and open only the ones we need.

Port control (snail.gnu.org and frog.gnu.org)
---------------------------------------------

On snail.gnu.org and frog.gnu.org there is no firewall. Instead, only services needed to provide the services that match the intended purpose of the machine are launched. All other daemons are de-activated.

Port control (yoda.gnu.org)
---------------------------

On yoda.gnu.org there is a firewall. rules are in file yoda.gnu.org:/etc/iptables.rules which are read from script yoda.gnu.org:/etc/init.d/packetfilter. Tese rules are written for netfilter (iptables).

Monitoring
**********

Netsaint
========

Netsaint was installed on http://snail.gnu.org/netsaint/. The configuration (/etc/netsaint/*) was created by hand. Neat was installed and can be used to fine tune the configuration.

This Netsaint instance is only supposed to watch over GNU machines.

Host groups were created for each hosting facilities so that different group of people can be sollicited if a problem occurs.

Contact : Cyril Bouthors

Mrtg
====

mrtg and addons have been installed with apt-get, two packages :

  * mrtg version 2.8.9-1

  * mrtgutils version 0.2

Some more scripts are used, they are located in /usr/local/bin, they are coming from http://mrtg.xidus.net/

  * swapstat.sh

  * stat.pl

Mrtg snail.gnu.org
------------------

  /etc/mrtg/mrtg.cfg was created by hand.

     Output dir is /var/www/mrtg/

  /etc/mrtg/mrtg-switch.cfg was generated with make -C /etc/mrtg mrtg-switch.cfg and should be regenerated in the same way whenever the switch usage changes.

     Output dir is /var/www/mrtg/

  /etc/mrtg/mrtg-storage.cfg was generated with make -C /etc/mrtg mrtg-storage.cfg and should be regenerated in the same way.

     Output dir is /var/www/mrtg/storage/

The /var/www/mrtg/index.html file was generated with:

     make -C /etc/mrtg /var/www/mrtg/index.html

The /var/www/mrtg/storage/<host>.html files were generated with:

     make -C /etc/mrtg /var/www/mrtg/storage/<host>.html

Contact : Cyril Bouthors, Rodolphe Quiédeville

Mrtg fr.fsf.org
---------------

Output dir is /var/www/fr.fsf.org/mrtg/

Contact : Rodolphe Quiédeville

Screen sessions
***************

When logged in as root on frog.gnu.org or snail.gnu.org, one can connect to a shared screen(1) session via screen -x. These screen sessions contain terminals connected to the switch, the powerboot, the serial console.  They are created at boot time by /etc/init.d/screen according to the content of ~root/.screenrc and use small shell scripts in /usr/local/bin to establish the connections. All sessions are logged permanently in /var/log/screen.

Although it is possible to talk to the switch or the powerboot without using the screen session, this is strongly discouraged because no log of the commands will be archived.

On all screen sessions, the escape character is C-\ instead of the default C-a so that editing stuff with emacs is not a nightmare.

A list of active windows is displayed permanently at the bottom of the screen session.

Most commonly used screen commands:

  C-\ d exit from the screen session (does not destroy it).

  C-\ 0 switch to terminal 0

  C-\ 1 switch to terminal 1

  C-\ 2 switch to terminal 2

  C-\ 3 switch to terminal 3

  C-\ c create a new terminal with a shell inside, exiting the shell will close the terminal.

  C-\ ? help

Crash recovery
**************

If snail.gnu.org crashes for some reason, do the following:

  * ssh -l root frog.gnu.org

  * screen -x

  * check error messages on the snail console terminal.

  * check error messages in /var/log/screen/snail.

  * switch to the powerboot terminal and toggle the snail powerswitch.

  * switch back to the snail console terminal and wait for the grub screen to show.