help any time. Move around with ls, cd, cat.
Edit a file with pico <file> — save with Ctrl-O, quit with Ctrl-X.
Then run the test command in your ticket to see if it worked.The game keeps it short. Here’s the real explanation of what each service did and how you configured it — roughly the way PowerNet actually ran, circa 1998–2002. (In the terminal you can also type man for any of them: useradd, apache, radius, bind, hosts, cron, sendmail, aliases, pop3, ftp, disksuite, frontpage.)
An ISP is a stack of small, single-purpose Unix daemons, each doing one job and mostly ignoring the others. PowerNet’s lived on Solaris x86 boxes (Intel hardware running Sun’s Unix — cheaper than buying Sun’s own SPARC iron):
• httpd (Apache) — web & customer site hosting
• radiusd (Livingston RADIUS) — checks dial-up logins. Livingston invented RADIUS, and PowerNet ran Livingston PortMaster modem banks, so this pairing was natural.
• named (BIND) — DNS, so names turn into numbers
• sendmail + qpopper (POP3) — sending and picking up mail
• cron — scheduled chores (log rotation, MRTG graphs)
• plus wu-ftpd (uploads), MRTG (bandwidth graphs), inetd/syslogd, and NTP
One thing that did not live on the Solaris box: Usenet news. PowerNet ran that on wally.uspower.net, an old Windows NT machine running DNEWS (NetWin) — an NT news server that pulled articles from the upstream feed on demand instead of taking the full firehose.
Before a customer can host a web site or get e-mail, they need an account on the server. One command creates it:
useradd -m -d /home/foxbait -s /bin/false foxbait passwd foxbait
-m makes their home directory (their web files live in /home/foxbait/public_html); -s sets the login shell; passwd sets the password they’ll use for FTP uploads and POP3 mail. That one account is the foundation everything else hangs off — web space, mailbox, and login.
Notice the shell is /bin/false, not a real one. Hosting and e-mail customers don’t need a Unix command prompt — they upload pages over FTP and read mail over POP3. Giving them a non-login shell lets both of those keep working while making sure they can’t telnet in and land a shell on your server. Fewer ways to break something; fewer ways to be broken into. (That’s how PowerNet set up most of its web-and-mail customers.)
That’s Ticket #1: create foxbait with -s /bin/false, set a password, and check it with finger foxbait. Gave them a real shell by accident? Fix it with usermod -s /bin/false foxbait.
One machine, one IP address, hundreds of customer sites. The trick is name-based virtual hosting. When a browser asks for a page it sends a header naming the site it wants:
GET /index.html HTTP/1.1 Host: www.foxvalleybait.com
Apache reads that Host: header and serves the matching <VirtualHost> block’s DocumentRoot. You switch the feature on once, then add one block per customer:
NameVirtualHost *:80
<VirtualHost *:80>
ServerName www.foxvalleybait.com
DocumentRoot /home/foxbait/public_html
</VirtualHost>
ServerName is the name to match; DocumentRoot is the folder of files to serve. To add a customer you create their folder, drop in their site, add the block, and restart Apache (apachectl configtest first, then apachectl restart).
The footgun: leave off the closing </VirtualHost> and Apache won’t start at all — every site on the box goes dark, not just the new one. Always configtest before you restart.
The customer’s modem dials the PortMaster (the Network Access Server). The PortMaster knows no passwords — it hands the username and password to radiusd as an Access-Request, signed with a shared secret only the two of them know. radiusd looks the user up in /etc/raddb/users:
jdoe Auth-Type := Local, Password == "winnebago"
Service-Type = Framed-User,
Framed-Protocol = PPP,
Framed-MTU = 1500
The first line is check items — what must be true to let them in (here, the password). The indented lines are reply items — instructions sent back to the PortMaster on success. Service-Type = Framed-User + Framed-Protocol = PPP mean “start a PPP internet session.”
Get the password right but forget the reply items and the customer authenticates… then drops in a few seconds, because the PortMaster was never told to bring up PPP. Two halves of one login. (RADIUS also logged session start/stop — accounting — which is exactly what the billing system, Rodopi, read to bill by the hour.)
DNS turns www.foxvalleybait.com into an IP address. To host a domain you run a master zone: named.conf points at a zone file, and the zone file holds the records.
zone "foxvalleybait.com" {
type master;
file "foxvalleybait.com.zone";
};
$TTL 86400
@ IN SOA home.powernetonline.com. host.powernetonline.com. (
2026062101 ; serial <-- BUMP THIS ON EVERY EDIT
10800 3600 604800 86400 )
IN NS home.powernetonline.com.
IN NS ns.powernetonline.com.
www IN A 207.0.55.10
@ IN A 207.0.55.10
The SOA opens the zone and carries the serial number. NS records say which servers are authoritative. A records map names (www, @ = the bare domain) to IPs. Convention for the serial is YYYYMMDDnn.
The classic mistake: edit the zone but forget to raise the serial. Secondary servers only copy changes when the serial goes up, so your edit silently never propagates and you spend an hour wondering why. Bump the serial. Every. Single. Time.
DNS is wonderful until it’s slow or down — and then your servers can’t find each other. So every Unix box also kept a tiny flat file, /etc/hosts, mapping names to addresses, one per line:
127.0.0.1 localhost loghost 156.46.191.20 home.powernetonline.com home 156.46.191.30 ns.powernetonline.com ns 207.0.55.20 wally.uspower.net wally
The format is just <IP> <full name> <short alias>. The resolver checks this file before it asks DNS (that order lives in /etc/nsswitch.conf as hosts: files dns), so an entry here resolves instantly and keeps working even when DNS doesn’t.
An ISP listed its own core machines here — the name servers, the mail box, and the news box — so internal jobs never waited on a network lookup. PowerNet’s Usenet server, wally.uspower.net, was an NT box down the hall; the nightly mail and news scripts referred to it by name.
The footgun in Ticket #5: if wally isn’t in DNS or /etc/hosts, every script that names it stalls ~30 seconds per lookup before giving up. One line in /etc/hosts — an IP and the name — makes the stall vanish.
cron runs commands for you at set times. Each line is five time fields and then the command:
# min hour day-of-month month weekday command 15 1 * * * /usr/local/sbin/rotate-logs # 1:15 AM daily 0 * * * * /usr/local/news/expire # top of every hour */5 * * * * /usr/local/mrtg/bin/mrtg ... # every 5 minutes
A * means “every.” The iconic ISP cron job is MRTG every five minutes — it polls the router’s byte counters and redraws those green/blue bandwidth graphs of your T1. It only draws a point when it runs, so if cron isn’t firing it, the graph flatlines.
Period gotcha: on Linux you write */5 for “every 5 minutes,” but old Solaris cron didn’t understand */5 — you had to spell it out: 0,5,10,15,20,25,30,35,40,45,50,55. The game accepts both and ribs you gently if you take the Linux shortcut.
sendmail moves email over SMTP. It works fine; its configuration is the legend. The live config, /etc/mail/sendmail.cf, is around 1,800 lines of dense rewriting rules that look like this and that nobody edits by hand:
R$* < @ $=w . > $* $: $1 < @ $2 . > $3 R$* < @ $+ > $* $#smtp $@ $2 $: $1 < @ $2 > $3
Instead you write a short, readable .mc file in the m4 macro language and compile it into sendmail.cf:
OSTYPE(`solaris2') DOMAIN(`generic') define(`confDOMAIN_NAME', `powernetonline.com') FEATURE(`access_db') MAILER(`local') MAILER(`smtp')
m4 ../m4/cf.m4 powernet.mc > /etc/mail/sendmail.cf
In the 1990s, open mail relays were an epidemic — spammers would bounce mail through any misconfigured server — and new sendmail releases shipped security fixes constantly. So you frequently built the latest sendmail from source. That’s Ticket #7, and here’s the whole dance:
cd /usr/local/src wget ftp://ftp.sendmail.org/pub/sendmail/sendmail.8.9.0.tar.gz tar xzf sendmail.8.9.0.tar.gz # unzip + untar in one step cd sendmail-8.9.0 sh Build # detects your platform, compiles everything sh Build install # installs the binary + a fresh sendmail.cf
That sh Build wrapper was sendmail’s gift to its admins: it figured out whether you were on Solaris, SunOS, Linux or BSD and ran the right make for you. Compiling your own mail server, from a tarball you fetched over FTP, to plug a spam hole — that is about the most authentically 1998 thing on this whole page.
newaliases: the one-step everybody forgetsAn alias points one mail name at another. Mail to a role address or a forwarding name gets rerouted to a real mailbox. You keep them in /etc/mail/aliases:
postmaster: root abuse: root webmaster: jamie sales: jamie # the new flyer address oldemployee: newperson@example.com
Left side is the name people write to; right side is where it actually goes (a local user, an external address, or a comma-separated list). It’s how you do postmaster@ and abuse@ (which the internet basically requires you to have), mailing lists, and “forward this person’s mail.”
sendmail does not read this text file when mail arrives — for speed it reads a compiled database (aliases.db). So editing the file changes nothing until you run newaliases to rebuild that database. Forgetting newaliases is the single most common “but I did add it!” moment in mail administration. That’s the whole point of Ticket #8.
Sending and receiving mail are two different services. sendmail accepts mail over SMTP and drops it into a mailbox file on the server. But the customer’s mail program — Eudora, Outlook, Netscape Mail — reaches in and downloads that mailbox over POP3, which on a 90s Unix box was served by qpopper.
POP3 is used in short bursts, so you didn’t run it as a heavy always-on daemon. Instead the inetd “super-server” listened on the port and launched qpopper only when someone connected. You switch it on with a line in /etc/inetd.conf:
pop3 stream tcp nowait root /usr/local/sbin/popper popper
…then make inetd notice the change by sending it a hang-up signal:
kill -HUP inetd
Lots of services worked this way (FTP, telnet, finger, POP3) — one listener, started on demand, instead of dozens of idle daemons eating a box with 128 MB of RAM. kill -HUP doesn’t kill inetd; the HUP (hang-up) signal is the classic Unix convention for “re-read your config file.”
The support call in Ticket #9 is the giveaway: a customer who can send mail but not receive it. Sending is sendmail/SMTP; receiving is POP3/qpopper — two separate switches, and only one of them was flipped.
Two different FTP jobs ran at an ISP. Hosting customers uploaded their web pages over FTP, logging in with their own account and landing in /home/<user>/public_html. And the ISP ran a public anonymous FTP archive — the dial-up installer, modem drivers, shareware — that anyone could download from without an account.
For most of the 1990s the standard was wu-ftpd (the Washington University FTP daemon), started on demand by inetd. It was everywhere — and infamous for a run of remote-root security holes around 1999–2000, so a lot of ISPs (PowerNet included) moved to ProFTPD, which ran standalone and had a clean, Apache-style config:
<Anonymous ~ftp>
User ftp
Group ftp
UserAlias anonymous ftp
<Limit WRITE>
DenyAll
</Limit>
</Anonymous>
The footgun — and it’s Ticket #10: that <Limit WRITE> DenyAll is the whole ballgame. Leave anonymous upload enabled and within days strangers discover your open FTP and fill it with pirated software and worse — on your bandwidth, in your name. Anonymous FTP is for downloads out, never uploads in.
By the late 1990s a lot of small-business sites weren’t hand-coded — they were built in Microsoft FrontPage, which had a Publish Web button instead of FTP. That button doesn’t speak FTP; it speaks HTTP POST to a set of CGI programs the FrontPage Server Extensions add underneath the customer’s web:
/home/foxbait/public_html/ _vti_bin/ shtml.exe _vti_aut/author.exe _vti_adm/admin.exe _vti_pvt/ service.pwd, service.lck, per-page metadata _vti_cnf/ _vti_txt/ _vti_log/
(_vti_ stands for Vermeer Technologies Inc., the company that wrote FrontPage before Microsoft bought it in 1996.) Microsoft actually shipped a Unix port of the extensions — maintained by Ready-to-Run Software — so Apache boxes like PowerNet’s could host FrontPage customers right alongside everyone else. You switch them on one web at a time with fpsrvadm.exe (the binary really is named .exe, even on Solaris):
fpsrvadm.exe -o install -p 80 -w /home/foxbait -xu foxbait -xg foxbait
The -xu / -xg flags set the Unix owner of those _vti_ folders, and that is the catch that generated a thousand support calls: the author.exe CGI runs as the customer’s own account. If the extensions are owned by root, the customer’s Publish button fails with a flat “cannot open service… access denied.” Own them to the customer (-xu foxbait -xg foxbait) — or fix it after the fact with fpsrvadm.exe -o chown … — and publishing just works. That’s the whole of Ticket #12. (The extensions were also a perennial security headache, which is part of why plain FTP and, later, real content-management systems eventually won out.)
Reliability wasn’t something you bought — it was something you built and then watched. PowerNet’s boxes ran Solaris 2.6 with Solstice DiskSuite 4.1 doing software RAID 1: two Seagate Barracuda 2 GB drives mirrored, kept cool at 62°F with a fan on each drive. (The Usenet news feed, meanwhile, came down from the upstream provider’s server, news.alpha.net.)
Mirroring a boot disk is famously fiddly — and we have the receipts. In July 1998, Jamie posted to comp.unix.solaris describing the exact trap: the data was mirrored, but pull the primary drive and the machine fell to single-user mode, “like the Master Boot record is not mirrored.” The fix (from a generous senior admin at Verio) is the lesson baked into Ticket #11: balance the metadb state-database replicas across both disks, run metaroot so root actually boots from the mirror, and write a boot block onto the second disk.
“I want to go on vacation this winter… I have been watching my equipment for 2 years straight now, about 16 hours a day. I want a hard drive to die and I want the machine to keep going like nothing ever happened.”
— Jamie Dolan, comp.unix.solaris, July 1998
Unix folklore says real admins use vi or emacs. Plenty of small-ISP admins just used pico — the plain, friendly editor that came with the Pine mail program. The commands you need are printed right along the bottom of the screen, where ^ means the Ctrl key:
^O WriteOut (save) ^X Exit ^G Get Help ^C show Cur Pos (line/column)
So: edit, press Ctrl-O to save, Ctrl-X to quit. On a phone with no Ctrl key, just tap those labels at the bottom of the editor — they work too.