This is the engine room. Underneath the friendly home pages and the rate cards, a small ISP was a stack of single-purpose Unix daemons, each doing one job. Here’s what each one did and what its configuration actually looked like — roughly the way PowerNet Online ran it on Solaris x86 boxes, circa 1998–2002.
Everything below is real and period-accurate, simplified for clarity (IP addresses and customer names are illustrative). Want to try it instead of read it? The same twelve jobs are playable in ⚡ PowerNet Boot Camp — a fake terminal where you edit these files and find out if it works.
One IP-connected box (usually several) running a handful of daemons.
PowerNet’s servers were Solaris on Intel hardware (Solaris x86) — the cost-smart move versus buying Sun’s own SPARC iron. On those boxes:
# the daemons that mattered, and what each one did httpd Apache web & customer site hosting radiusd Livingston checks dial-up logins (PortMaster modem banks) named BIND DNS — turns names into numbers sendmail sendmail sends & routes mail (SMTP) popper qpopper hands mail to Outlook/Eudora (POP3, via inetd) cron cron scheduled chores (log rotation, MRTG graphs) mrtg MRTG draws the T1 bandwidth graphs proftpd ProFTPD FTP: customer uploads + the anonymous download archive inetd inetd the "super-server" that starts small services on demand syslogd syslog the central log
The one exception: Usenet news did not live on the Solaris stack. PowerNet ran news on wally.uspower.net, a Windows NT box, using DNEWS — covered below.
The mental model: a customer pays you, and that single act touches the whole stack — RADIUS lets them dial in, BIND + Apache serve their web site, sendmail + qpopper carry their mail, cron keeps the graphs honest, and DiskSuite keeps the whole thing alive when a disk dies. The rest of this page walks the stack one daemon at a time.
The very first thing a signup needed: an account on the box.
Before a customer could host a site or get e-mail, they needed an account on the server. That one account is the foundation everything else hangs off — their web space, their mailbox, and their login. You created it with one command:
as rootuseradd -m -d /home/foxbait -s /bin/false foxbait passwd foxbait
-m makes their home directory (their pages live in /home/foxbait/public_html); -d sets where it is; -s sets the login shell; and passwd sets the password they’ll use for both FTP uploads and POP3 mail. A disk quota kept any one account from filling the drive.
/bin/falseLook at that -s /bin/false. A hosting/e-mail customer doesn’t need a Unix command prompt — they upload pages over FTP and read mail over POP3, and both of those work fine without a login shell. Setting a non-login shell means the account can still do its job but nobody can telnet in and land a real shell on your server.
The footgun: give a customer a real shell (/bin/sh) out of habit and you’ve handed a stranger an interactive account on the machine that runs your whole town’s Internet. PowerNet gave most web-and-mail customers no shell — fewer ways to break something, fewer ways to be broken into. (Fix a slip with usermod -s /bin/false foxbait.)
One server, one IP address, hundreds of customer web sites.
The trick is name-based virtual hosting. A browser names the site it wants in the request:
GET /index.html HTTP/1.1 Host: www.foxvalleybait.com
Apache reads that Host: header and serves the matching block’s files. You turn the feature on once, then add one block per customer:
NameVirtualHost *:80
<VirtualHost *:80>
ServerName www.foxvalleybait.com
DocumentRoot /home/foxbait/public_html
</VirtualHost>
ServerName is the name to match; DocumentRoot is the folder to serve. To add a customer you create their folder, drop in their site, add the block, then apachectl configtest and apachectl restart.
The footgun: leave off the closing </VirtualHost> and Apache won’t start at all — every site on the box goes dark, not just the new one. Always configtest before you restart.
How a customer’s modem actually got online.
The customer’s modem dials a Livingston PortMaster (the Network Access Server). The PortMaster holds no passwords — it forwards the username and password to radiusd as an Access-Request, signed with a shared secret. The secret is configured on both ends:
portmaster1.powernetonline.com s3cr3t-shared-key
Then each subscriber is a record in the users file:
/etc/raddb/usersjdoe Auth-Type := Local, Password == "winnebago"
Service-Type = Framed-User,
Framed-Protocol = PPP,
Framed-MTU = 1500
The first line is the check items (what must be true to let them in). The indented lines are reply items sent back on success: Service-Type = Framed-User + Framed-Protocol = PPP mean “start a PPP internet session.”
The footgun: get the password right but forget the reply items and the customer authenticates… then drops in a few seconds, because the PortMaster was never told to bring up PPP.
PowerNet detail: Livingston invented RADIUS, and PowerNet ran Livingston PortMaster banks — so this pairing was the natural one. RADIUS also logged session start/stop (accounting), and that’s exactly what the billing system, Rodopi, read to bill by the hour.
Being the authority for a domain — and the #1 footgun.
To host a domain you run a master zone: named.conf points at a zone file, and the zone file holds the records.
zone "foxvalleybait.com" {
type master;
file "foxvalleybait.com.zone";
};
/var/named/foxvalleybait.com.zone
$TTL 86400
@ IN SOA home.powernetonline.com. host.powernetonline.com. (
1998061501 ; serial <-- BUMP THIS ON EVERY EDIT
10800 3600 604800 86400 )
IN NS home.powernetonline.com.
IN NS ns.powernetonline.com.
www IN A 207.0.55.10
@ IN A 207.0.55.10
The SOA opens the zone and carries the serial number. NS records say which servers are authoritative. A records map names (www, @ = the bare domain) to IPs.
The footgun: edit the zone but forget to raise the serial. Secondary servers only copy your changes when the serial goes up — so your edit silently never propagates and you lose an hour wondering why. Convention: YYYYMMDDnn. Bump it every single time.
An ISP also ran a reverse zone (in-addr.arpa) for its IP block, so an address could be turned back into a name with a PTR record. Mail servers checked it: no reverse record, and your mail often got rejected as spam.
10 IN PTR www.foxvalleybait.com. # 207.0.55.10 -> the name
Name resolution that works even when DNS doesn’t.
Long before (and alongside) DNS, every Unix box kept a flat list of names → addresses. The resolver checks this file before it asks DNS, so an entry here resolves instantly and keeps working even when DNS is slow or down.
/etc/hosts127.0.0.1 localhost loghost 156.46.191.20 home.powernetonline.com home 156.46.191.30 ns.powernetonline.com ns 207.0.55.20 wally.uspower.net wally
The format is just <IP> <full name> <short alias>. The lookup order lives in /etc/nsswitch.conf as hosts: files dns — “check the file first, then DNS.”
PowerNet detail: the core machines — the name servers, the mail box, and the NT news box wally — went in here so the nightly mail and news scripts never stalled on a network lookup.
Doing things on a schedule — and a genuine Solaris gotcha.
cron runs commands at set times. Each line is five time fields then a command:
/var/spool/cron/crontabs/root (edited with crontab -e)# min hour dom mon dow command 15 1 * * * /usr/local/sbin/rotate-logs # 1:15 AM daily */5 * * * * /usr/local/mrtg/bin/mrtg ... # every 5 minutes
A * means “every.” The iconic ISP job is MRTG every five minutes: it polls the router’s byte counters and redraws those green/blue bandwidth graphs of the T1. It only draws a point when it runs, so if cron isn’t firing it, the graph flatlines.
Period gotcha: on Linux you write */5 for “every 5 minutes,” but old Solaris cron didn’t understand */5 — you had to spell it out:
0,5,10,15,20,25,30,35,40,45,50,55 * * * * /usr/local/mrtg/bin/mrtg ...
The mail server everyone feared — and how you compiled it.
sendmail moves email over SMTP. It works fine; its configuration is the legend. The live file, /etc/mail/sendmail.cf, is ~1,800 lines of rewriting rules nobody edits by hand:
R$* < @ $=w . > $* $: $1 < @ $2 . > $3
Instead you write a short, readable .mc file in the m4 macro language and compile it:
powernet.mc → m4 → sendmail.cfOSTYPE(`solaris2') define(`confDOMAIN_NAME', `powernetonline.com') FEATURE(`access_db') MAILER(`local') MAILER(`smtp')
m4 ../m4/cf.m4 powernet.mc > /etc/mail/sendmail.cf
In the 1990s, open mail relays were an epidemic and new sendmail releases shipped security fixes constantly — so you built the latest from source:
cd /usr/local/src wget ftp://ftp.sendmail.org/pub/sendmail/sendmail.8.9.0.tar.gz tar xzf sendmail.8.9.0.tar.gz # unzip + untar in one step cd sendmail-8.9.0 sh Build # detects your platform, compiles sh Build install # installs the binary + sendmail.cf
That sh Build wrapper figured out whether you were on Solaris, SunOS, Linux or BSD and ran the right make for you. Compiling your own mail server, from a tarball you fetched over FTP, to plug a spam hole — about the most authentically 1998 thing on this whole page.
Pointing one mail name at another — and the one step everyone forgets.
An alias reroutes mail: role addresses, mailing lists, and forwarding a departed employee’s mail all live here.
/etc/mail/aliasespostmaster: root abuse: root webmaster: jamie sales: jamie # the new flyer address oldemployee: newperson@example.com # forward off-site
Left side is the name people write to; right side is where it goes (a local user, an external address, or a comma-separated list). This is how you provide postmaster@ and abuse@ — which the internet essentially requires you to have.
The catch that bites everyone once: sendmail does not read this text file when mail arrives — for speed it reads a compiled database (aliases.db). Editing the file changes nothing until you run:
newaliases
How customers picked their mail back up.
Sending and receiving mail are two different services. sendmail drops mail into a mailbox on the server; the customer’s mail program (Eudora, Outlook, Netscape Mail) downloads it out over POP3, served by qpopper.
POP3 is used in short bursts, so you didn’t run it as a heavy always-on daemon. The inetd “super-server” listened on the port and launched qpopper only when someone connected:
/etc/inetd.conf# name socket proto wait user server args
ftp stream tcp nowait root /usr/sbin/in.ftpd in.ftpd -l
telnet stream tcp nowait root /usr/sbin/in.telnetd in.telnetd
pop3 stream tcp nowait root /usr/local/sbin/popper popper
Then you make inetd notice the change by sending it a hang-up signal:
kill -HUP inetd
The classic support call: a customer who can send mail but not receive it. Sending is sendmail/SMTP; receiving is POP3/qpopper — two separate switches, and only one of them was flipped. (And kill -HUP doesn’t kill inetd — HUP is the Unix convention for “re-read your config.”)
Two jobs: customers upload their web pages, and the public downloads from an archive.
FTP did double duty at an ISP. Hosting customers uploaded their pages over FTP — logging in with their own account and landing in /home/<user>/public_html. And the ISP ran a public anonymous FTP archive — the dial-up installer, modem drivers, shareware — that anyone could pull down without an account.
For most of the 1990s the standard was wu-ftpd (the Washington University FTP daemon), started on demand by inetd. It was everywhere — and notorious for a run of remote-root holes around 1999–2000, which pushed a lot of ISPs (PowerNet included) over to ProFTPD: it ran standalone and had a clean, Apache-style config.
<Anonymous ~ftp>
User ftp
Group ftp
UserAlias anonymous ftp
<Limit WRITE>
DenyAll
</Limit>
</Anonymous>
The <Anonymous> block maps the anonymous login to the unprivileged ftp user, jailed in its home directory. Customer uploads, by contrast, just use their normal account — which is exactly why those accounts get a non-login shell: FTP still works without a shell.
The footgun: that <Limit WRITE> DenyAll is the whole ballgame. Leave anonymous upload enabled and within days strangers find your open FTP and fill it with pirated software — on your bandwidth, in your name. Anonymous FTP is downloads out, never uploads in.
Mirroring the boot disk so the box survives a dead drive (software RAID 1).
DiskSuite mirrors two drives so the server keeps running when one dies. Three pieces must line up — and the third is the one everyone misses:
# 1) state-database replicas on BOTH disks (they vote on what's current) metadb -a -c 3 c0t0d0s7 metadb -a -c 3 c0t1d0s7 # 2) a submirror per disk, joined into one mirror device d0 metainit -f d10 1 1 c0t0d0s0 metainit d20 1 1 c0t1d0s0 metainit d0 -m d10 metattach d0 d20 # 3) make it actually BOOTABLE off either disk metaroot d0 # rewrite vfstab/system: root boots from the mirror installboot .../bootblk /dev/rdsk/c0t1d0s0 # write a boot block to disk 2
The real bug: skip step 3 and your data is mirrored but the box still won’t boot off the survivor — pull the primary drive and it falls to single-user mode, “like the Master Boot Record isn’t mirrored.”
This one is true: in July 1998 Jamie posted that exact problem to comp.unix.solaris — a server on Solaris 2.6 with DiskSuite 4.1, mirroring two Seagate Barracuda 2 GB drives. The thread, and the wish behind it (“I want a hard drive to die and the machine to keep going like nothing ever happened”), survive in the early-years page.
The one service that wasn’t on the Solaris stack at all.
A 1990s ISP was expected to carry Usenet news — thousands of discussion groups customers read with Outlook Express, Free Agent or tin. The generic Unix answer was INN (innd), but PowerNet didn’t run it on the Solaris boxes.
PowerNet detail: news lived on wally.uspower.net, a Windows NT machine running DNEWS (NetWin). DNEWS was the popular small-ISP choice on NT because it could pull articles on demand — fetch a group’s posts from the upstream feed the first time someone asked for it, and cache them — instead of swallowing the entire multi-gigabyte-a-day Usenet firehose and the disks that would have required.
The upstream feed itself came down from the provider’s server, news.alpha.net. So the architecture was: alpha.net (the big upstream) → wally (PowerNet’s NT/DNEWS box) → the customer’s newsreader. Conceptually, a DNEWS feed config looked like:
# DNEWS: pull from the upstream, fetch groups on demand newsfeeds = news.alpha.net pullnews = on expire = 4 # days to keep articles before reclaiming disk
Running news on a separate NT box was a very real small-ISP pattern: Usenet was disk- and bandwidth-hungry and politically noisy (spam, cancels, binaries), so you fenced it off from the machines that carried your customers’ logins, web sites and mail.
The smaller daemons that rounded out the box.
ufsdump nightly level-0 backup of /home to a 4mm DAT tape in /dev/rmt/0 (cron, 2 AM),
rotated OFF-SITE each morning — the backup you never restore is just a rumor
TCP wrap. tcpd + /etc/hosts.allow / hosts.deny — telnet & finger fenced to the staff network
IP Filter ipf packet firewall (/etc/ipf/ipf.conf): block inbound, pass only our services
quotas per-user disk limits on /home (rq in /etc/vfstab) so one account can't fill the disk
Webalizer chews Apache logs into the monthly traffic graphs customers always asked about
MRTG the bandwidth graphs themselves (cron runs the poller; see above)
Majordomo mailing lists (announcements, support) driven through sendmail
syslogd central logging to /var/log/messages
xntpd keeps the clock right (NTP) — mail, RADIUS accounting + logs depend on it
inetd starts FTP / telnet / POP3 on demand (see POP3 above)
ns the SECOND box — a slave/secondary DNS server (ns.powernetonline.com),
so the domain still resolves even if home is down
None of it is glamorous, and that’s the point: a working ISP was a dozen small, well-behaved programs, each doing one job, watched over by one or two people who knew where every config file lived. Now you do too.
→ Ready to try it? ⚡ Play PowerNet Boot Camp — edit these exact files in a fake terminal and see if it works, or if you broke the whole box.