Building a website on SDF

Contents

• Introduction
  • Getting Started
• Your First Page!
• Bot Traffic and Robots.txt
• Problems
• Diagnose

Introduction

Creating your own websites can be fun and SDF gives webspace to all its validated users, so the moment you become validated you can begin creating your own websites straight away.

In this tutorial when text looks like this this means it's a command you can run in your shell, once you've logged into SDF.

Getting Started

• mkhomepg - This command needs to be executed before you can start building your web pages.

  When you execute this program, it will create a directory in which you can put all your web files. This directory is sym-linked into your home directory, under the name 'html'. (A sym-link is sort of like a shortcut in Microsoft Windows.)

  You will be prompted to select a URL for your website. This URL is what other internet users type into their web browser to view your web pages. If you're not an ARPA member then you may only use url's in the form of '<your login>.freeshell.org'. For instance, my web pages are stored at 'alterego.freeshell.org'. If you're an ARPA member, there are many different domains you may use.

Your First Page!

Creating your first page is SO easy! First you'll need to open a text editor. SDF has many text editors installed but new users to UNIX often find that nano is the easiest to work with.

• nano ~/html/index.html - This command starts the nano text editor, creating a new file, called 'index.html' in your html directory.

Once you've executed the command above. You'll notice that the terminal layout has changed. If you press keys on your keyboard you'll notice they appear in the "text buffer". Write out the text below into this program.

<html><head>
  <title>My First Page!</title>

</head><body>
  <h1>My First Page!</h1>

  <p>This is my first html web page!</p>

</body></html>

Now that you have copied the text out into 'nano', you need to save the file. To do this, hold down the 'CTRL' key on your keyboard and whilst pressing that, press the letter 'o'.

Right, now, do you remember that URL you selected when you ran 'mkhomepg'? Open a web browser and type that into the address bar. Hopefully, after a few seconds. You should see your very own web page!

Bot Traffic and Robots.txt

Mainstream search engines and AI-crawlers will honor a robots.txt file in your website root directory. Wikipedia is a good starting point for information on robots.txt, but due to the large number of potential bot user agent names, a suggested robots.txt that only allows a small number of small-web friendly bots is as follows:

User-agent: Ariadne
User-agent: Wibybot
User-agent: MojeekBot
User-agent: DuckDuckBot
User-agent: search.marginalia.nu
Disallow:

User-agent: *
Disallow: /

This example allows only the five search bots listed, and blocks anyone else. Here is some info about the listed user agents:

• Ariadne is the bot for Clew search (https://clew.se/about/).
• Wibybot is for Wiby search (https://wiby.org/about/).
• MojeekBot is for Mojeek search (https://www.mojeek.com/about/).
• DuckDuckBot is for the well-known duckduckgo.com search engine.
• Search.marginalia.nu is for Marginalia search (https://about.marginalia-search.com/).

If you find bots in your access logs that are clearly ignoring robots.txt, here are some ideas:

• For private CGIs or pages, don't link to them and give them non-standard names, or put them behind basic auth.
• For public sites or pages, consider using static site generators, or just hand-crafted HTML and CSS, rather than dynamic blogs or wikis. This won't prevent the bot traffic, but will at least lessen the load on the web and database hosts.

Problems

Diagnose

$ grep abc123.sdf.org /var/log/httpd/*.log

$ less /meta/log/abc123.sdf.org

For aditional help, check the information in the FAQ page "What About the Access Logs, Statistics, and Quotas?"