National Capital FreeNet Annual Report for 1993 Hardware/Software/System Design Andrew Patrick (aa118) Ian Allen (aa610) Feb 1 1994 "We have not succeeded in solving all of our problems. In fact, the solutions we have found have served to raise a whole new set of questions. In some ways, we are as confused as ever; however, we feel we are confused on a higher level and about more important things." (source unknown) General ------- The hardware, software, and system design aspects of the FreeNet have been done with the assistance of many volunteers. It is through their efforts that the FreeNet has gotten this far, and I want to begin by thanking them. It is also important to recognize the contribution made by Carleton University, not only in housing the equipment, but also all the volunteer hours their staff have put in to installing and operating the system, and teaching us naive operators how to do things right. We must also remember that we would not have any hardware to operate if it were not for our corporate sponsors. Most of the computer and communications equipment we use has been donated by corporations who share the vision of building a FreeNet. Finally, the most significant event this year was the hiring of a full-time contractor for the System Administration duties. Ian Allen has made the system hum, and squeezed blood from a stone as we grow and grow. The addition of Ian has certainly improved the FreeNet a lot, and made my life a lot easier. Hardware -------- Computer Equipment ------------------ The FreeNet host machine was donated by Sun Microsystems before the opening last February. Later in the year, Sun provided an upgrade to the system so that it now contains two 50-Mhz SPARC processors. This makes it one of the best computers available, and this has allowed us to grow very rapidly. As we grew, it became clear that mail and news traffic was becoming a significant load. Did you know that the NCF is often one of the top 10 Usenet posting sites in the world, and that is not counting our local discussion groups? Carleton University allowed us to experiment with moving our news and mail traffic onto one of their machines, and this improved things a great deal. Sun came through again with a donation of a second SPARCstation. This machine is currently being used for the mail and news systems, and the Carleton machine is running the AGM vote system. At the end of the year, Resudox Online Services donated a SPARC 2 style machine, in recognition of the FreeNet's role in bringing an awareness of the Internet to the Ottawa area. This machine, called "freenet2" is currently being prepared for use alongside the other machines. A memory upgrade is needed before it is fully functional. Future Issues ------------- In the near future we will have the following configuration: - main Sun SPARC 10/512 computer (for users) - SPARC 2 style computer (for users) - Sun SPARC IPX computer (for mail and news) The issue we are wrestling with now is getting all these computers to operate together in a seamless environment where it does not matter which computer you are connected to. We will also be faced with acquiring new equipment as we outgrow even this configuration. What equipment should we try to get? Can we once again get donated equipment? Also, we are soon going to have to invest in disk space and more memory for the current equipment. The more long-term issue is how long we can continue to grow like this. How far does such a configuration scale? We are already experiencing problems because of the size of our user list. What happens when it doubles to 30,000 and we are running 8 machines? What happens when all the mail for 30,000 users converges on the one machine that handles the mail? Telecommunications Equipment ---------------------------- Here again it is he generosity of our corporate supporters that has allowed us to make such progress. Gandalf Technologies has supplied a large number of modems for the FreeNet and we have grown from about 4 lines at the beginning to 96 lines, with more on the way. During this time, we have installed two terminal servers to concentrate all these lines into the network ports on the computer systems, and we will soon have to add a third. Each time we have added more lines, we have also had to deal with growing pains. The computer system often was not prepared for the increase in load, and we had to scramble to adjust things and make them work. It also became clear that we could not just keep adding lines and expect the FreeNet to work, so we began also acquiring the additional computer equipment that was needed. We also began moving the phone number to a new number. The City of Ottawa has offered us space on their CENTREX service, which will significantly reduce our phone charges. We expect that all the lines will be moved to the new number, with one line left to direct users, within the next few weeks. Future Issues ------------- The issues we are facing include acquiring more equipment, including modems and terminal servers. What equipment should be try to get, and what cost/benefit analysis should we be looking at? In the longer term, we must always be on the lookout for opportunities to reduce or eliminate our telephone charges, since these represent such a large portion of our budget. Thus, new technologies like interactive Cable TV and digital radio may be the ultimate solution for the FreeNet, and we must always be aware of developments in these areas. Software -------- In the software area, the main task this year has been to install and operate the FreePort software that is the heart of the system. Also, in addition to the core menu navigation routines, FreePort includes a number of supplemental programs such as the Vote module and the IRC service. These supplemental services were installed this year with the help of volunteers. Our use of the FreePort software revealed a number of problems and 'bugs' in the routines. Much of the effort this year was spent repairing these problems. When Ian started work, he was able to make significant progress in fixing the most persistent problems, and the system is running relatively error-free at this point. A 'freenet-tech' mailing list has been created to share experiences and expertise with other FreeNets around the world. Another of Ian's tasks this year has been to start the process of making the system fully bilingual, working towards making it multilingual. This area has seen a lot of progress and, with the assistance of volunteers, we now have large portions of the FreeNet available in French and English. We have also made significant progress in adding services that are not part of the FreePort system. For example, the Gopher Internet navigation tool has been installed and is proving to be quite popular. Future Issues: -------------- There is a long list of problems and 'wish items' in the 'TODO' area of the Administration menus (see the Appendix). Many of these items are important, but they have had to be delayed as higher priority items are dealt with. We hope that some of these items can be assigned to volunteers, and that most of them can be cleaned up in the next year. The coordination of volunteers will be a major task in the upcoming year. There are qualified people in the FreeNet community who are able to contribute to developing or maintaining software. Some of those people have been working as volunteers and been instrumental in getting the NCF to where it is today, but there are others who we have not had the time to coordinate. With the help of the Volunteer Coordinator, we hope to be able to get more people working on the system software. Another major task will be to complete the bilingual aspects of the system. This will again require the help of volunteers since every program that prints to the screen will have to be modified to support languages other than English, and this is a very large job. In the mean time, we have virtually all the tools necessary to make FreeNet handle any combination of PC, Macintosh, and ISO Latin-1 character tables. What remains is the time to wrap these tools in user-friendly menus and provide reasonable defaults. In the longer term, serious consideration must be given to the question of how long we stick with the FreePort system. FreeNets who are just getting organized are looking at using Gopher or WWW (World Wide Web) systems as the basis for their operation, and this may be a major trend for the future. We will have to examine what the benefits are to going to a more modern system, and what the costs and implications will be. It is likely that we will try these systems out and work towards new technologies as they become reliable. System Design ------------- The main activities in the area of system design have been to set-up the menus as new information is added. This has been quite an ad-hoc procedure as new information is slotted into the existing menus, or new categories are added where needed. To this point, little planning has been done, which is probably appropriate given the age of the system -- there is little use in planning when we don't know how the system will be used and how it will evolve. Nevertheless, we do seem to have a fairly well organized and very rich system, especially when compared to the other FreeNets. Future Issues: -------------- There are a number of system design issues that have come up in the past few months, such as the organization of the main menu, the integration of bilingual information, and the use of menu 'tags' to indicate the type of menu item. However, there is no mechanism in place to deal with these issues, and it is not appropriate for each one of these to be debated by the Board. An active System Design committee will be an important goal for the next year, where decisions on these issues can be made and given either to the System Operations team for implementation or the Board for approval, depending on the nature of the issues. This Design Committee should address the following issues, as a start: - reorganization of the main menu - develop a rational for menu placement - evaluate a scheme for identifying menu types - implement a style guide for menu design - consider the issue of integration with advanced services, such as Gopher and WWW Appendix: Administration TODO List ================================== National Capital FreeNet TODO List ================================== Last updated approximately: Sun Jan 30 04:25:23 EST 1994 -IAN! These are some of the problems we know about, and some of the things we and others think might need doing some day. It's also a request to you to volunteer if you have the skills to help us fix any of these problems. The Usenet News group ncf.sub.hardware-software is for discussions related to these items, or to ask questions about these items. Discussions on broader design issues happen in ncf.sub.system-design. Everyday questions and gripes about the system should be posted to ncf.admin. Priority for attention by NCF staff is determined more-or-less by the order in this file. Priority for NCF volunteers is determined more-or-less by what the volunteers what to contribute. If you want to work on one of these problems, let us know. Most of these problems require both C language programming skills and a knowledge of the Unix software environment. If you think something is broken and it isn't on this list, please let us know; you often find problems before we do. - Create a secure voting environment for the Annual General Meeting. Set up third Sun machine in secure mode to support this. [in progress: -Ian! 93/12/12] - Write AGM SysAdmin summary report for NCF Board. [in progress: -Ian! 94/01/24] - Configure, compile, and install the tcpwrappers suite. To protect NCF against unwanted IP connections. [94/01/26] - Stop the newsreader from dumping core. (see /usr/tmp/core*.nr) [in progress: -Ian! 93/12/01] - Stop menu.c from dumping core. (see /usr/tmp/core*.menu) [in progress: -Ian! 94/01/10] - Fix mr nonsense: Enter Command: p Checking for new mail...no new mail. rewriting mail file write message: getc: Error 0 write of message 0 failed: Error 0: possible mailbox truncation expected to write 4607378 chars, wrote 0 chars expected to write 1000 messages, wrote 0 messages Unsuccessful update of /freenet/home/37/ab913/mailbox: Error 0 Mailbox unchanged. closemailfile: cannot rewrite mail file /freenet/home/37/ab913/mailbox: Error 0 Error closing mail file, trying to restore...restore unsuccessful, possible loss - Stop telnetd from dumping core and leaving junk in utmp. (see /usr/local/src/telnet.filter/telnetd/CORE/core) Clean dead junk entries from /etc/utmp daily. [in progress with Marc aj313: -Ian! 93/12/02] - Redo the NCF online registration procedures to make them faster and less prone to errors. Maybe move the whole lot to another machine? Remove passwords from master log files. Rewrite procedures; get new text translated into French and installed. [in progress: -Ian! 93/12/03] - Move the current online registration procedures to the less loaded freenet2 machine. Fix SunOS bugs regarding multiple updates of the NIS maps (makedbm) corrupting the maps. Provide a script to do "vipw" editing and locking on the /etc/yppasswd file on freenet2. [in progress; mostly done: -Ian! 94/01/10] - Prepare NCF for several-computer operation. Pick machine for daemons and recompile everything to use name. Fix account creation software to use new NIS passwd file. Write a script to partition terminal server lines. Fix bbwho, etc. [Some of this is started; more to be done -Ian! 93/12/20] - Create software to send out thank-you notes to NCF donors. [93/10/15] - Install new Pico-3 that lets you select an output file. In final stages of preparation by Kamil Buchler ac476. [94/01/10] - Install news group hierarchy for software team. Keep the software/hardware volunteers informed and active. Probably use ncf.sub.hardware-software as our discussion area. [in progress -Ian! 93/12/13] - Decide on a Language Preference profile format for NCF users. Discussions should happen in ncf.sub.system-design and/or ncf.sub.hardware-software. [in progress with Marc aj313: -Ian! 93/12/12] - Implement a translation scheme from ISO Latin-1 to various PC and ASCII formats to allow French to be read on old and dumb terminals. Write a menu interface that lets users choose their translations. [Done; testing now: Marc E. Gauthier aj313; see also "go cfranco"] Needs a better user interface, plus a script to display various byte combinations that correspond to various translation schemes to let users pick the scheme that looks best. [94/01/11] - Write a script/program to surround file-transfer programs to turn off the telnetd character translation for the duration of the transfer. Has to handle interrupts gracefully. [Colin ac534 is working on this; 93/12/12] - Investigate using Unix groups instead of common-user xx* accounts for multi-user access to info. (Also needs support scripts that keep group write turned on!) Setting up groups and setting file ownership and group write can be done manually at present. [94/01/01] - Find out why processes get hung on annex ports. "cat -v -u /dev/ttyxx" seems to clear them out. [93/11/01] - Document why people must not subscribe to LISTSERV lists on NCF. Set up a way for people to know that they can read these via the culist.* news hierarchy. [94/01/14] - Install MUSH environment for SysAdmin so mail can be sorted and tracked better. (Got the files from Andrew now.) [to be done -Ian! 93/12/01] - Get a bug reporting and tracking system installed to keep track of what volunteers are working on what. UW has something; GNU has something. [research in progress -Ian! 93/11/15] - Collect and assign volunteers to read the ncf.* groups and answer problem questions, especially in ncf.admin and ncf.general Create an online "Help Desk". [93/11/01] - Help Marita Moll get her Academy One gopher area set up. - Create a system for handling NCF announcements. Likely a clone of the UW read_system_news feature. Maybe modify menu.c to put a reminder on the screen when an unread announcement is pending? - Go over all /freenet/rootdir/logs/menulogs/errors and tell info providers about the errors in their menus. [maybe Lawney aj486 would do this? 94/01/01] - Install the UW "lockeditor" package that locks a file and edits it. This is essential in an environment with multiple users all editing the same stuff. Marc and I are using patched UW binaries for this at the moment since we don't have the UW software environment installed that would let us compile from source. I tried to get UW to feed us their software environment; but, their Sun feed machine is in a more restrictive part of the computing environment and a feed would be "counter to UW executive policy". Sigh. [to be done -Ian! 93/12/23] - Analyse modem usage statistics to detect bad modems. [Chris Hawley was looking at this informally... 93/12/14] - Modify my "modemzero" script to be more verbose and useful for novices. Merge stats info with phone numbers. [94/01/01] - Install gcc [Fetched by Colin ac534; waiting installation by Ian! 93/12/18] - Add a write() time-out to menu.c so that the program stops hanging on write() calls on stuck pty's. Every time you send it a signal, it gets stuck writing to the closed terminal... [94/01/15] - Install the npasswd programs to ensure people choose good passwords. Also run the passwd cracker to eliminate obvious weak points. [Fetched by GJHagenaars aj247; waiting installation by Ian! 93/12/13] - fix NCF catdb not to complain about empty .db* files [in progress by GJHagenaars aj247 93/11/30] - Put a wrapper around the "nr" newsreader and/or modify it so that it requires explicit action by the user to be able to post to non-ncf (open Usenet) news groups. [in progress: -Ian! 93/11/18] - Make a mail alias to which someone may mail and get the User Agreement by return mail. Make a general "info-request" mail alias. [in progress: Colin ac534 and GJ aj247 93/11/30] - Make /usr/local/{man,bin,src} consistent and orderly. Install the man pages for stuff present in /usr/local/bin/ Set up a stock .cshrc fragment that sets PATH and MANPATH correctly. [93/12/01] - Set up funny-money usage accounting for user login time. [was in progress by Cindy ak288 93/11/24; but, she ran out of time] - Some volunteers think they can build a "rack" for stand-alone fast modems (e.g. sportster) using PC hardware. We need CSA approval. [in progress: Chris ah654 Rick al904 93/12/23] - Write a C subroutine that will nicely "abbreviate" a Full Name to any arbitrary column width. Useful for "who" listings and any place a user name has to appear in a fixed size space. [in progress: Robert ak032 93/12/23] - Collect together NCF "oral tradition" and make a FAQ [in progress: Basil aa591 is curious about this 93/12/23] - Roll the various FreePort log files regularly. Write a script to pick out essential messages from /usr/adm/messages and mail these to people as they happen. - Roll the Annex terminal server log files regularly. - Kermit does not seem to do binary transfers [is this still true?] - Keep disk usage statistics, either via quota or the stuff in /usr/lib/acct/ Notify users who have too much stuff online. See Andrew's scripts in $fsrc/carleton-src/disk_usage/ as a starting point. Also see $fsrc/carleton-src/dusage/ (from UW) Port the UW "meander" script that looks for old files and mails users about their imminent deletion. - fix forwarding to allow X.400 addresses fix mailbox reader to parse X.400 as real messages. [Andrew and I have some data on this; need a volunteer. -IAN!] - add reply-all command to e-mail system - install mail gateway information (how to get mail to various systems and networks) - check on cross-posting of messages [what does this item mean? -IAN] - create a "public" download area where Freenet software and other "goodies" can be distributed. Candidate items here are: - the "Mu" Freenet Menu Emulation Program - terminal packages pre-configured to settings known to work on the Freenet - character set translation templates ... Also need to install the support programs that symlink or copy the files. [Matthew aa673 is working on this 94/01/15] - look at compression schemes to assist download times [the ToolBox has some. We need more. -IAN!] - develop a program to report system and menu usage statistics [the ToolBox has some. We need more. -IAN!] - Fix annex ports not to respond to triple-plus modem escapes sent back from the Unix side in the data stream. May require an "expect" script run at boot time to do this. - Find better solution to board mail archives (request by Dave Hughes). aa098 and aa099 - Reslolve opening login screen clutter. Logo or no logo? Terminal size info or not? Summary of ncf.announce items? - Dealing with the line sponsors information at login Resolve these conflicts: - Some Board members want the sponsorship line to stand out, in case sponsors might feel their sponsorship message isn't getting across; they want most other login status information suppressed. - However, the presence of this status information has cut user complaints about terminal size and scrolling down to almost nil; removal of the information may cause user frustration again. [Matthew aa673 has a program that boldfaces lines 94/01/15] - Fix mail forwarding so userid,userid doesn't send mail into real spool directory (rather than via m2mbox) where user can't get at it. - Create means for an NCF user to put special header lines in news articles so the user can post FAQs, etc., and be a real Usenet moderator. - Create restricted versions of various Unix programs Look into installing vi, ftpd, emacs, gopher, trn, etc. via chroot rather than by doing source modifications. [I'm working on this but not very hard. -IAN!] - Get some kind of real database software for NCF to handle all the accounts and password info. Jean Wilmot mentioned that Oracle might be interested? Andrew Patrick to follow up? [93/12/03] - Install the RFC931 Identity Service so our machine is identified to remote sites. The rfc931-users mailing list: rfc931-users-request@kramden.acf.nyu.edu - make .plan/.project file access via BBedit May not be useful now that NCF fingerd uses FreePort PMDB files. [in progres: Matthew aa673 94/01/15] - Make French commands work in menu.c e.g. time -> temps ou heure ou minutes [in progress little by little -IAN! 94/01/01] - Mail fixes to FreePort software back to Chet at CWRU Unsorted miscellaneous stuff to do: - after modifying terminal characteristics, prompt the user to save them - install Sun resolver routines patch - merge John's BBmenutest script with BBmenu - find a way to make FreePort documentation available to software voluneers - why does Veronica send back articles with escaped text? Fix bogus hilighting in searched and returned gopher/veronica text. The escapes don't work on some people's ttys. - allow user to edit CE config file to save desired options - modify the mail reader to have a command that will make a mail alias out of the current mail message address - restrict info providers to some subset of the file system Prevent IPs from browsing private work directories of other people. Needs work setting up a chroot environment. - flow control on "screen" broken -- why? Install latest version of screen and see what happens then. - screen should be setuid root to properly grab tty entry for security - add way for gopher telnet users to change escape from ^] - fix menu.c to allow a choice of logos at login (including none) - fix menu.c to "exec" processes rather than fork /bin/sh - make a menu.c command to mail stty parameters and environment to sysadmin - fix the exising menu.c command that tries to do this - plot numeric userid against last logged in: graph - split "last logged in" and "mail forwarded" permissions so that users can grant permission for others to see one but not the other - make a tar utility for work directory - make "VT100" synonymous with "vt100", or issue a warning - write a script to clean the work directory of the guest account - make a who list that uses your mail aliases for who is online - fix mail aliases so that they expand transitively properly A chain of a->b->c->d aliases fails to expand all the way if put in the alias file with a->b before b->c before c->d. - fix mail alias file to allow RFC822 addresses, e.g. with full names - make nr replies use Newsgroups line so posting threads don't split up - make a collected index of all menu titles a la gopher - index of recent stuff too so people know what's changed - find out if the News filesystem is full/down - script to monitor file system status - get "sysmon" from UofWaterloo - config sendmail to allow mail to news groups - config sendmail to allow mail to full name - allow editing .bcc file in home directory This would let users automatically keep copies of all their mail. - add Cc line to mailer: #define CC_PROMPT requires a vote to see if users want this, or maybe a .menurc item - make a tool that configures your terminal; it verifies and sets what your delete, kill, and interrupt characters are and saves that info in the .menurc. - put uuencode and uudecode in file tools - install menu area listing all known Internet providers in Ottawa UUNET, Resudox, Fonorola, etc. - who listing that abbreviates full names with some smarts so it will fit in 3 columns - fix FreePort database routines not to chop at 256 - strlen($USER) - does /etc/utmp have to be world-writable? getlogin() security? - fix the documentatin for Chet's Editor (CE) so that it reflects what's really possible (no shell escapes, etc.) - keep bugging OCTranspo for their bus database - ISC (Andy Reddick) wants/wanted NCF usage pattern data - remove permissions from all /work/ so shell users can't snoop user files - get Andrew Patrick's "chat" software installed to answer NCF questions Andrew says maybe Spring 1994? - install bozo filter for mail and "kill" files for news I have the code from CWRU for the bozo filter, but it's awful. - make "go" commands select menu lines/selections as well as menus - add better topic/thread searching to news reader - make it possible to save any file that the pager prints, into your work directory - make "fav" mode show number of articles and optionally only show groups with unread news - give users unlimited login time as long as modems are free - add modem stats file tweaks to .login files in shell accounts so we have a full set of login stats for modem use in $ffiles/modems/ - have UPS confirm your user before starting the editor - fix UPS and work directory file permissions to 0700? Files should be general read; /work should not be. - bundle all HELP files into a ZIP archive for PC users - write a program that displays different menu files depending on date/time - fix the file truncator to warn when it chops a user's file Use it on .signature files that are too long. - prevent arbitrary Gopher access to telnet sites Mike Richardson has an untested PERL wrapper for bbtelnet. - make mail reader ask for confirmation of huge delete requests (requested by Al Seaman) - mail reader seems to make old messages unread and new messages read at random. Hard to reproduce... - write .forward alias checker that prevents people from forwarding their mail to q@freenet.carleton.ca