root/tags/REL3_5/CHANGES

Copyright (c) 2006-2007, Joseph B. Kowalski
Portions Copyright (c) 2007 Kasimir Gabert
See LICENSE for licensing information

TorStatus - Tor Network Status Changes

===============================================================================
Version 3 Changes {{{
===============================================================================
--Version 3.5--
* Complete interface redesign
* Integrated operating system detection in by default
* Bandwidth change to have a progress bar
* Legend created
* WHOIS query integrated for IP addresses
* Background colors for different router status
* Overall softer colors
* Added notifications for Asc or Desc sort orders

--Version 3.4.2--

* (ticket 12) Fixed error with server IP detection for non-Squid servers
* (ticket 12) Added config option for Squid server

--Version 3.4.1--                             (changes made by Kasimir Gabert)

* Fixed bottom headers for new domain
* Updated IP detection so it works through Squid
* Added known mirrors bar

TODO:
* Move the two changes into configuration items
* Centralize the known mirrors and make it dynamic

--Version 3.4--                            (end of changes by Joseph Kowalski)
Rename of the DNSBL server to DNSEL (DNS Exit List) server. This was by
recommendation of Roger Dingledine of the Tor project. All filenames, variable
names, etc, have been adjusted to reflect the rename.

Added basic hit counting functionality to the DNSEL server. Now, every five
minutes, the DNSEL server will write a record to the "DNSEL_LOG" table with a
timestamp, a TotalResponses counter, and counters of how many "NOERROR",
"SERVFAIL", "NXDOMAIN", and "NOTIMP" responses it has sent out since it was
started. This procedure runs in it's own thread, and so should not have any
measurable effect on performance.

Improved the code that detects whether a user is accessing the site via a Tor
hidden service. Previously, I checked to see if the user was coming from
127.0.0.1. Now, I check to see if the "HOST" header provided by the client
upon access exists in the "$Hidden_Service_URL" from the config file. If so,
we tell them they are accessing the site as a hidden service. If not, we
proceed on to the normal checks to see if they are accessing through the Tor
network.

Modified the DNSEL server to return SERVFAIL instead of NXDOMAIN for requests
which are outside of the zone the DNSEL server is authoritative for.

Modified the DNSEL server to return NXDOMAIN instead of NOTIMP for requests
other than type 'A' and 'ANY' when the request is in the authoritative zone.

Modified the documentation / default variables for the DNSEL server to use the
"ip-port.torhosts.example.com" domain, rather than "tor.dnsel.example.com".
This encourages users to setup servers that are consistent with the
"torel-design.txt" document maintained by the Tor project.

--Version 3.3--
Complete rewrite of the DNSBL server in Java. This provided a performance
increase on the order of approximately 5x over the old Perl version of the
server. On a Pentium 2.4 GHz machine with 2 GBs of RAM, I was able to
consistently obtain load test results of 200 queries/sec or more. These tests
were conducted across the internet, and not on the local machine.

Addition of the "dnsbl_server.php" page. This page provides usage instructions
for the DNSBL server, and informs users of it's domain name. Note that the
link to "dnsbl_server.php" from the main page will only display if you
set the "$DNSBL_Domain" variable in the "config.php" file, so you can leave
this variable set to 'null' if you are not running a DNSBL server, or do not
wish to advertise it on the web site.

Addition of the "$Hidden_Service_URL" variable to the "config.php" file. If
you are also making the TorNetworkStatus site available through a Tor Hidden
Service, you can enter the URL here, and a link will be added to the main page.
If you are not making the site available as a Hidden Service, or do not wish
to advertise it on the web site, you can leave this variable set to 'null'.

Removal of the "tor_exit_status" page. This functionality is already provided
by the main page (Tells a user if they are accessing the site through the Tor
network or not), and the user already has two options to do custom exit
queries if desired: The "tor_exit_query" page, and the DNSBL server. Both are
linked to from the main page.

Modifications to the "tns_update.php" file and the database create script to
accommodate the re-write of the DNSBL server.

--Version 3.2--
Removed the first type of query that the DNSBL server could perform, which
was to simply lookup if a provided IP address was an active Tor server or not.
This was causing confusion amongst some users, and causing people to focus on
whether or not an IP is a Tor server at all, which is exactly what we are 
trying to get away from by providing an answer to the more granular question
"Is this IP address a Tor server that is capable of exiting to my destination
IP and port?", which is precisely what the second (and now only) query type
does. This had the added benefit or making the server a little faster, since
a bit of logic was removed. Please see the "README_DNSBL" file in the source
distribution for full setup and usage information.

--Version 3.1--
Nearly a complete redesign of the DNSBL server added in the last version to
improve performance. Major changes:

1) Previously, there was a Perl script which performed the actual listening
for DNS requests, and when one was received, the script passed on the details
of the request to a PHP script, which is where the main processing happened.
The PHP script would then return a value to the Perl script indicating success
or failure of the request. This had the horrible performance penalty of
needing to invoke PHP every time a new request was received. Now, the entire
process is handled within the same Perl script -- All of the PHP code to
process the DNSBL request has been ported over to Perl -- So there is no
calling an external executable again and again. Additionally, Perl runs faster
than PHP with all other factors being equal.

2) With the old PHP script, since it had to be invoked anew from within the
Perl script with each request, this also meant that a new database connection
had to be setup with each request. This also caused a performance penalty.
Now, since everything is handled from within a single script, a single
database connection can be maintained the entire time the script is running.

3) Previously, when the DNSBL script did it's database queries to pull the 
IP addresses and exit policies of Tor routers, it did it the same way that the
main application does, which means joining two large tables together on the
router fingerprint key. While this works fine for the main application and
it's very large queries, it did not suit the performance needs of a DNS
server. Now, I have added two new in-memory database tables specifically for
the DNSBL functionality. These two tables are populated during the update
cycle, and contain only the necessary information (prejoined) required by the
DNSBL server. The fact that they are in memory and are a lot smaller than the
other tables should help the DNSBL server to return queries much quicker.
These two tables act just like the set of NetworkStatus and Descriptor tables
in that only one is ever active at a given time. The update cycle will always
be updating the non-active table in the background while requests can continue
being served by the active table. When the update is done successfully, a flag
in the Status table is toggled to tell the application to begin using the
other table as the active one, and the process repeats -- No downtime for
updates.

Fixed a bug that would cause the DNSBL server to only return "NXDOMAIN"
responses if the database connection was lost until the script was restarted,
even if the database connection came back up later. Now, the server will
return "SERVFAIL" responses while the database server is down, but it will
resume correct operation once the database server comes back up.

Various other minor bug fixes, mainly relating to the "Tor Exit Status" and 
"Tor Exit Query" functionality that was added in the last release.

--Version 3.0--
Complete redesign of the way that the application determines if you are 
accessing the site through the Tor network or not, or if the provided IP 
belongs to a Tor server capable of exiting to the specified IP and port (See 
section on "Tor Exit Query" page, below). Previously, we would simply check to
see if the IP address that the user was coming from was a Tor server with the
'Exit' flag set to true, and tell them they were coming through the Tor
network if so. Now, we check if the user is coming from the IP address of a
Tor server, regardless of it's 'Exit' flag status. If so, we pull the Exit 
Policy for the matching Tor server(s), look at what IP and Port the client is
accessing us on, and parse the exit policy to determine if the Tor server(s)
would allow or deny such an access. We then give the user information on each
Tor server that matches their exit IP, letting them know if it would allow
access to us or not considering it's exit policy. Finally, we also tell the
user if it appears that they are coming to the site via a Tor Hidden Service
or from the local machine directly. This is determined by checking if the user
is coming to the site from the '127.0.0.1' IP address. Note that the
application now handles situations where there are multiple Tor servers with
IP's that match where the user is coming from (Or the IP provided, in the case
of the "Tor Exit Query" page, see below). In these cases, it will cycle through
each applicable Tor server to provide Exit Policy match information.

Addition of "Tor Exit Status" page. This page provides the same information
about whether a user is accessing the site through the Tor network as is
available on the main page, but without all the other information. This may be
easier for someone who just wants a simple answer for "Is my Tor installation
working?", and should serve as a good entry point for such situations. This
page also contains a link to the main Tor documentation page.

Addition of "Tor Exit Query" page. This page allows a user to enter in an IP
to check if it matches that of one or more Tor servers, and, if it does,
checks to see if those Tor servers are capable of exiting to the destination
IP address and port that the user provides.

Addition of a DNSBL server. This allows a standard, non-web interface method
for clients to query whether or not an IP address belongs to an active Tor
server, and whether or not an IP address belongs to an active Tor server that
would allow exiting to a specified destination IP address and port. Please
see the "README_DNSBL" file in the source distribution for full setup and
usage information.
===============================================================================
}}}
===============================================================================

===============================================================================
Version 2 Changes {{{
===============================================================================
--Version 2.0--

Small performance enhancement to background update process. When clearing old 
data from the descriptor and network status tables, we now use the "truncate 
table" command rather than the "delete from" command. This is faster because 
"truncate table" deletes all records from the table at once, while "delete 
from" deletes records one row at a time. 

Enhancement of "Require Flags" custom query option so that the user now has 
three options available for each flag: "Off", "Yes", and "No". "Off" means 
the status of the flag is not a factor in the search. "Yes" means a router 
must have flag set in order to show up in result set. "No" means that a router
must not have flag set in order to show up in result set. Previously, a user 
only had the equivalent of the "Off" and "Yes" options. Also, when a flag is 
set to "Off", it's column header in the result list will have a blue 
background, "Yes" will have a green background, and "No" will have a red 
background. This further increases the level of flexibility with searches.

Addition of "Country Code" field to "Tor Server Providing Network Status 
Opinion" section.  

Addition of background color to drop-down lists and edit box in advanced 
search section.

Minor changes to text in custom query section to facilitate clarity.


--Version 2.1--

Addition of "Aggregate Network Statistic Graphs / Details" page. This page 
contains four graphs of network-wide aggregate information: Number of Routers 
by Country Code, Number of Routers by Observed Bandwidth, Number of Routers 
by Time Running, and an aggregate summary of various criteria including router
flags.

Addition of "EventDNS" and "Hibernating" statistics to newly renamed 
"Aggregate Network Statistics Summary" section on main page.

Improvement to the way that the "Aggregate Network Statistics Summary" section
of the main page gets it's data. Previously, each piece of information used 
was collected by a separate SQL query. Now, all of this has been joined into 
one large query to cut out the overhead of multiple SQL calls.

Change to CSS which affects the highlight color of links which are on a white 
background, for increased readability.


--Version 2.2--

Fixed a bug in the code that detects whether or not the user is connecting to
the server through the Tor network. Previously, it would tell a user they were
connecting through the Tor network if their IP address matched the IP address
of any Tor server. Now, it will only tell the user they are connecting through
the Tor network if their IP address matches the IP address of a Tor server AND
that Tor server is an exit node.

Addition of exit node only CSV file download, for those who don't want the 
IP's of all Tor servers.

Addition of graph of exit routers by country code to Network Detail page. This
is useful for letting us see where we are most likely to exit the Tor network.

Addition of "BadDirectory" flag information everywhere we make use of flags. 
The "BadDirectory" flag was added in Tor version 0.1.2.5-alpha.

Minor cleanups to documentation to enhance readability.

Fixed minor bug in SQL database create script.


--Version 2.3--

Redesign of the "Custom / Advanced Query Options" box on the main page. I am
now using drop-down lists for the sort request, sort order, and address mode
parameters instead of radio buttons, which takes up a lot less screen real-
estate and also makes adding new options for these parameters in the future
more practical.

Slight changes to the "Application Server Details" (Previously "Application
Server Information") box on the main page to enhance readability.

Addition of the "Number of Descriptors In Cache" field to the "Application
Server Details" box.

Movement of the "Application Server Details" box to the bottom of the page. 
This has the dual purpose of improving page readability as well as making the 
"Approximate Page Generation Time" figure more accurate, since it is now
calculated closer to the bottom of the page. Previously, the PHP code which
generates the "Custom / Advanced Query Options" box still ran after this time
was calculated.

Update of source URL at bottom of page to new domain.


--Version 2.4--
Modified "Aggregate Network Statistic Summary" box to also show "percent of
total network" values for each statistic listed. This allows you to easily see
what percent of Tor routers are exit nodes, what percent are mirroring the
directory, what percent of the total network your current result set makes up,
etc.

Modified "Network Status Opinion Source" box to allow clicking on the nickname
of the router to jump directly to its router detail page. This makes it easy
for a user to get detailed information about the source of the network data
that they are seeing.

Modified code that detects whether a user is accessing the site through the
Tor network or not. Now, if the user is accessing through the Tor network, the
name of their exit node is displayed in addition to the IP address. Also, the
user can click on the name of the exit node to jump directly to its router
detail page. This makes it easy for a user to get detailed information about
the exit node they are currently coming through.

Modified page footer to also include a link to the change history and license
of the application.

Removed the "$MinRouterThreshold" variable from the config file, and all code
that made use of the variable. This really became obsolete when version 1.4 of
the application was released and we started using two sets of tables in the
database for storing router information instead of one. Since that time, a
user will always have their page generated from the active set of router
tables in the database, while the next refresh operation is being performed on
the inactive set. This makes this variable, which was used to see if a
database refresh was in progress and display an error to the user back when we
used a single set of tables, obsolete.

Fixed a bug in the generation of the Comma Separated Value (CSV) list of exit
nodes. A comma was incorrectly being added after the last entry in the file,
which might have caused problems for parsers.

Added "Number of Routers by Platform" graph to network detail page. This lets
us easily see how popular different operating systems are for running Tor
routers.

Modified "Number of Routers by Time Running (Weeks)" graph on network detail
page to be full page width. This makes this graph easier to read since it
usually has a lot of items along the x axis.


--Version 2.5--
Fixed lots of minor HTML and CSS errors. All pages now validate as "CSS
version 2.1" and "HTML 4.01 Transitional".

Modified the handling of all SQL queries which return a result set so that the
result set populates into an associative array rather than a numerically
indexed array. This allows items in the result set to be referred to by column
name rather than a numerical array index. This is a significant step towards
being able to offer the user the ability to do things such as select which
columns they want displayed on the page, etc, as it is much more flexible
than using numerical indexes because your entire result set handling code is
no longer affected by re-arranging the order of items in your query, or adding
and removing items on the fly. It is also generally considered a database
programming best practice, as a lot of confusion and potential for bugs is
eliminated by being able to use easily recognizable column names as opposed to 
numerical indexes.

Modified the "Address Display Mode" in the "Custom / Advanced Query Options"
so that users now have a third choice -- "Hostname + IP Address". Previously,
the user could only choose one at a time, and had no way to view both
simultaneously. "IP Address Only" mode is still the default as the "Hostname +
IP Address" mode can result in a much larger amount of data being returned,
but now the option is there for those who want it.


--Version 2.6--
Added the ability for users to select which columns they want displayed in the
main router table. This should be good for ensuring that the page does not 
grow larger and larger as new informational items about Tor routers become 
available, since we can just maintain reasonable default settings and leave
the obscure stuff off unless a user intentionally turns it on. Also, the SQL
query for obtaining the main router list has been made dynamic so that it will
only request columns from the database if the user will actually be displaying
them.

Added "Fingerprint" as a column display option.

Removed "Address Mode" from "Custom / Advanced Query Options" since it is made
obsolete by the users ability to select which columns they want displayed,
which offers more granularity.

Efficiency improvement in the way that it is determined if a given flag
column for a given router should display the red "x" or green check mark.
Previously, there was an "If" statement that set the appropriate CSS class for
the table data cell based on whether a flag variable for that router was '0'
or '1'. Now, there is a CSS class that ends in '0' and '1', and the variable
value is just dumped directly into the class statement in the appropriate
place. This saves up to '$NumberOfDifferentFlags X $NumberOfRoutersInResultSet'
'If' statements from being executed on each page load, normally many thousand.

Fixed a bug in the generation of CSV files that was causing a comma to be
added after the last column in a row. According to RFC 4180, this is incorrect.
Also, added a newline character (\n) to the end of files.

Added "Comma Separated Value (CSV) List of Current Result Set" option. This
lets a user get a comma separated value (CSV) file of their current search 
result set, exactly as displayed on screen (Including sort preferences, column
display preferences, and any other custom query parameters that may be
applied). The first record (row) in the generated CSV file will contain the
column names for reference. This should be very useful for users wishing to
export the data from a custom query into an external data source.


--Version 2.7--
Moved the column display options to their own page, rather than having them on
the main page in the "Custom / Advanced Query Options" box. You can get to the
column display options by following the "Custom / Advanced Display Options"
link from the top of the page and then clicking on the "Column Display
Preferences" link. This should provide a much more comfortable interface for
customizing column display options.

Added the ability for users to re-configure the order that columns display in. 
Previously, users could only add and remove columns. This should allow much
more flexibility to users wishing to customize the look of the page. 

Updated "Comma Separated Value (CSV) List of Current Result Set" functionality
to support the newly added ability of users to re-configure column display
order. This way, the CSV file of the current result set benefits from the same
level of flexibility as the main page display.


--Version 2.8--
Added the ability for users to display four additional columns on the main 
page: "Platform", "Contact", "EventDNS", & "Hibernating". Users can sort by
the new columns and, in the case of the "EventDNS" and "Hibernating" items,
filter query results based on whether they are true or false, as they are
treated as "Flags" by the application. All four of the newly added columns are
set to "Off" by default.

Updated the "Comma Separated Value (CSV) List of Current Result Set" script to
support the addition of the new columns.

Bug fix to the "Comma Separated Value (CSV) List of Current Result Set" script:
Commas (,) and double-quotes (") are now replaced with a dash (-) and a single-
quote ('), respectively, when they occur in a field to be written to the CSV
file, as these characters would cause problems for automatic parsers.

Modified the "Router Detail" page to display information on the "EventDNS" and
"Hibernating" items alongside the other flags, since it has now been
determined that the application will be treating these items like any other
flag. Previously, these two items were displayed in the "General Information"
section of the page, not the "Router Flags" section.

Modified the order in which some items display in the "Aggregate Network
Statistic Summary" section of the main page and the "Aggregate Summary -- 
Number of Routers Matching Specified Criteria" graph on the "Network Detail"
page to enhance consistency throughout the application.

Minor source code cleanups to enhance consistency between scripts.


--Version 2.9--
Various application-wide code cleanups to fix all known situations that would
generate PHP warnings.

Fixed a rare division by zero bug that could occur during the generation of
the bandwidth history graphs on the router detail page.

Updates to the way "Uptime" information is presented. Previously, the "Router
Detail" page would calculate the real-time uptime of the router (By looking
at the difference between the current time and when the router descriptor was
last published, and adding this value to the published "Uptime" value in the
descriptor, compensating for time zone differences), but the other places that
made use of the "Uptime" value did not, such as the main page, the "Network
Detail" page, and the "Comma Separated Value (CSV) List of Current Result Set"
functionality. These places would simply use the static "Uptime" value from
the latest descriptor, and would not perform any real-time calculations on it.
Now, all places that make use of the "Uptime" value perform the real-time
calculations on it. This eliminates discrepancies between what the "Router
Detail" page shows and what is shown everywhere else. Additionally, code to 
gracefully deal with situations where a certain descriptors publish time is
obviously wrong, such as when it's in the future, was added to all places
doing the "Uptime" calculations.

Modified all places that display a "DirPort" value to say "None" rather than
"0" for routers that do not mirror the directory. This is less confusing.

Modified all places that display a "CountryCode" value to say "N/A" rather than
nothing for routers where country code information could not be retrieved.
Again, less confusing.

Made the "Tor Network Status" text at the top and bottom of each page a link 
to get back to the main page. This should be more user-friendly than having to
hit "back" to get to the main page when on the "Network Detail" and "Router 
Detail" pages. The links also exist on the main page itself, which will just
refresh the page with the user's current settings.

Added two new array variables to the config file to allow the default column
display options to be set on a global scale. These options are what will be
used when the user has not yet performed any customization on their display
preferences. Both the active/inactive columns and their display order can be
specified.

Changed default column display preferences -- Added "Hostname" and removed
"Bad Dir" and "Bad Exit".

Removed support for the "EventDNS" descriptor flag entirely. The logic for this
functionality was faulty, and since all new Tor versions have EventDNS enabled
by default, and the Tor developers plan to remove the old DNS logic entirely
at some time in the future (According to the "dir-spec" document), I decided
to just go ahead and remove this now since it is becoming irrelevant.

Removed "IRIX" as a listed operating system on the "Platform Graph", as it
seems there have been no Tor servers running on this OS for some time now.

Other minor fixes/enhancements.
===============================================================================
}}}
===============================================================================

===============================================================================
Version 1 Changes {{{
===============================================================================
--Version 1.0--

Initial release.


--Version 1.1--

Addition of router detail page. 

Addition of real-time-calculated router uptime to router detail page.

Addition of bandwidth graphs to router detail page. This functionality makes 
use of the free 'JPGraph' PHP libraries.

Addition of country code data to main and router detail pages. This 
functionality makes use of the free 'GeoIP' PHP API and country database.

Separation of database refresh operation from main page render operation. The 
database refresh operation has been moved out of the web root into it's own 
script, 'tns_update.php'. This greatly increases page render speed in 
situations where the local-cache is expired.

Addition of 'tns_agent.php' script to handle background scheduled database 
refresh operations.

Movement of the SQL database creation script out of the web root.

Addition of ability to click on a column header on main page to sort. Multiple 
clicks alternate sort direction.

Addition of code to handle situations where a user attempts to load a page 
while a background database refresh is in progress. In these situations, the 
user will be notified of the database refresh being in progress, and the page 
will automatically reload in 10 seconds, retaining any sort or search criteria
that was initially passed by the user.

Various performance improvements during main page render, some major. Code 
cleanups. Implementation of CSS / HTML restructuring to greatly reduce the 
size of page.


--Version 1.2--

Application-wide security fixes. 

Multiple XSS vulnerabilities fixed.

Possible SQL injection vulnerability fixed.

Code cleanups, further performance enhancements. 
                                                

--Version 1.3--
                       
Major changes to the way the application gets it's network status information 
and router descriptors. 

Previously, for network status information, the application would cycle 
through the local Tor server's "cached-status" documents of Authority servers 
you explicitly allowed in the application config file, using a different one 
for each local-cache refresh. This had two problems. First, it meant that you 
would get a different number of routers listed, and oftentimes different 
properties listed for a specific router depending on which Authority server's 
cached-status doc had been used. Second, and this is the bigger of the two 
problems, is that this is not the way the Tor network works. The Tor network 
leaves it up to each client/server to calculate what it believes to be the 
correct view of the network based on the multiple sources of network status 
information it has available. This is important for security reasons, and 
plays a significant role in making it harder for someone to play some bad 
tricks on network users. So, now network status information is obtained by 
asking the local Tor server's control port to list all routers it has an 
opinion about. This ensures the data being presented has been through the 
proper "voting" process done by the Tor software, and is what the local Tor 
server believes to be correct.

Previously, for individual router descriptors, the application would do one 
of two things. One, the default, was to directly ask whichever Authority 
server was used for the last local-cache refresh. This was bad, as it 
basically meant that for the duration of the current cache life, every 
visitor going to the "router_detail" page was causing a request for a 
descriptor to be sent to that Authority server. Not what we want for the 
Authority servers, I'm sure. Two, a user could specify that they always wanted
to use a certain Tor server for descriptor requests. This fixed the issue with
hammering the Authority servers, but only if people used the option. Now, 
router descriptors are also obtained through the local Tor server's control 
port. Also, router descriptors are no longer written to files on the hard 
drive. This was unnecessary and slow.

These changes allowed a lot of clean-up to be done in the config file. There 
are a lot less variables in there now.

 
--Version 1.4--

Addition of an option to export the IP Addresses of all current Tor routers to
a CSV (Comma Separated Value) file. This may be of some use for people looking
to get a quick, current list of Tor router IP's, whether to formulate block 
lists, or to assist with other Tor network status apps / pages.

Addition of "Bandwidth" and "Uptime" columns to main page. The bandwidth 
reading is obtained from the "Bandwidth Observed" parameter of the router 
descriptor and is displayed in KB/s. The uptime reading is obtained from the 
"Uptime" parameter of the router descriptor and is displayed in days. Both of 
these columns are sortable.

Back end changes to make the addition of the "Bandwidth" and "Uptime" columns 
to the main page, described above, possible. These changes consisted of making
the scheduled background update process request and parse the descriptors of 
every router that the local Tor server has an opinion about, as identified by
it's network status request response, so that the bandwidth and uptime 
information (and everything else in the descriptor) could be inserted into the
database at this stage to be available for use on the main page. Previously, 
descriptors were only requested and parsed on an as-needed basis when a user 
loaded the "router_detail.php" page, and the data parsed from them was not 
stored. Now, it's all inserted into the database during the main update 
process which opens up the possibilities for all kinds on new stuff. The 
downside to doing it this way is that the main update process CAN take 
considerably longer (especially when the local Tor server is very busy), which
led to the second major back end change: Using two router (local-cache) tables
instead of one. Basically, only one table is marked as active at any given 
time, and all page loads use the active table. When the update process runs, 
it runs on the non-active table, and marks it as the active table upon 
successful completion. This prevents users from seeing the "Database refresh in
progress" error message, as they will instead just be seeing the old data 
until the new data is ready. I was quite serious about not causing a 
performance hit to page load times due to the new functionality, and this was
the solution implemented to avoid that.
          

--Version 1.5--

Major performance enhancements to both the background update process and the 
main page load process.

First, for the update process, we are now obtaining the full network status 
opinion document and the full recent descriptor document from the control port
of the Tor server. Previously, we were obtaining the full network status 
opinion document, and then, as we parsed out this file, we requested server 
descriptors of the Tor server's control port individually. This tended to 
sometimes cause somewhat of a load on the Tor server and cause the descriptors
to be fed back very slowly, sometimes taking up to a half hour for a full 
refresh. Now, since we grab both full documents all at once, it only takes the
Tor server a few seconds to provide them. Second, we are now writing the 
parsed network status information and the parsed descriptor information into 
two separate database tables (NetworkStatus & Descriptor). This makes for 
cleaner design and tables with less columns. And, in keeping with the spirit 
of the updates made in Version 1.4, there is two of each of these tables 
(NetworkStatus & Descriptor) so that one set can be updated while the other 
set can continue to be used by folks loading the page and running queries, 
etc., until the new set is ready.

Performance improvements to the load time of the main page were obtained by 
switching to the MySQL MyISAM storage engine, instead of the InnoDB engine we 
had been using previously. MyISAM is significantly faster than InnoDB, and, 
since we currently don't make use of transactions, there is no reason not to 
use it. Also, indexes were added to the NetworkStatus and Descriptor tables 
on the "Fingerprint" column, since this is what is used for the join of these
two tables during page load. This further increased the query speed.

Some other small performance improvements were implemented, especially on the
"router_detail.php" page.

The update process now has more robust error handling.

Addition of "Count" column to main page.

Addition of green check marks and red x's to indicate whether a flag is "Yes" 
or "No". Previously, it would just print "Yes" or "No".

Modifications of font selections and background images to hopefully achieve 
two things: Fit more in a smaller space, and make the page easier to read.


--Version 1.6--     

The application now performs reverse-DNS lookups during the update process and
stores this information in the database. This enables a user to choose whether
they would like addresses displayed as IP or Hostname. Of course, whichever 
display they choose, it is sortable.

Added Hostname information to the router detail page as well.

Added "Last Update Elapsed Time" field to main page. Good way to see if your 
system is experiencing problems during the background updates.

Added "Current Descriptor Signature" field to the main page for the Network 
Status Source router information section. 

Minor CSS changes to improve layout and formatting in certain situations.  


--Version 1.7-- 

Changes to the way variable handling is done. I moved all state maintaining 
variables into PHP session, such as all of the "Flags" variables and the 
Address Display Mode, SortRequest, and SortOrder variables. Also, wherever 
possible because of addition to session, variables were removed from the URL's
in the anchor tags of the generated page. The only ones that were left in 
URL's were variables that need to be manipulated by clicking on a link, namely
the SortRequest and SortOrder variables. This makes the URL's in the anchor 
tags of the generated page much shorter and cleaner, since a whole lot less is
being passed around via GET. This in turn reduces the generated page size, 
sometimes considerably depending on the custom query options in effect. 
Another nice benefit to doing things this way is that if a user has custom 
options set, completely browses away from the site, and later comes back 
within their same browser session, all of their options will be configured as
they were when they left.

Also, variables that needed to remain in URL's have shorter names to further 
reduce the amount of data being passed around via GET, and hence the length 
of the URL's in the generated page.

Further cleanups to HTML formatting and CSS on both the main page and the 
router detail page.

Addition of timer to bottom of main page to indicate how long page generation
took on the server.

 
--Version 1.8--  

Changes / cleanup to CSS. All CSS information is now defined in one external 
style sheet which resides in the "web/css" directory, and the different parts 
of the code that need the style sheet simply reference it rather than defining
everything at the top of every HTML page output to the browser.

Addition of row highlighting for Mozilla Firefox / Opera. This makes it easier
to visually track which row you are currently moused over. This feature does 
not currently work in Internet Explorer since that would require JavaScript, 
and I'm trying to keep this app JavaScript free for those who browse with it 
disabled.

When sorting by IP Addresses, they are now sorted naturally, rather than as 
strings. This should make the IP Address search quite a bit more meaningful. 
This change also applies to the CSV (Comma Separated Value) file list of IP's.


--Version 1.9--  

Added proper escaping of problem characters to all strings that get populated 
from data parsed from router descriptors and network status documents before 
database inserts are performed. This should prevent the update process from 
dying when these characters are encountered, which would sometimes happen. 

Now using the standard "gethostbyaddr" PHP function to do hostname lookups. 
Previously, I was doing some custom parsing on the output of the Unix "host" 
command, as I thought it would be faster, but, after comparing the two, I see 
no speed degradation with using "gethostbyaddr". The advantage of using 
"gethostbyaddr" over "host" is that we are not using a platform-specific 
command which limits where the application can run without modification. 

Addition of Advanced Search functionality, which lets you perform free-text 
searches on the following data: Fingerprint, Router Name, Country Code, 
Bandwidth, Uptime, Last Descriptor Published, IP Address, Hostname, Onion 
Router Port, Directory Server Port, Platform, & Contact. You can apply the 
following modifiers when searching the above data: Equals, Contains, Is Less 
Than, or Is Greater Than. The Advanced Search works in conjunction with the 
other Custom Query options (Sort By, Sort Order, Address Display Mode, and 
Require Flags), allowing a great deal of flexibility to customize the result 
set as desired. 

Addition of indexes where necessary in the database to better facilitate 
speedy free-text searches.

Addition of "Hostname" field to "Tor Server Providing Network Status Opinion" 
section.

Addition of feature to tell user if they are accessing the page through the 
Tor network or not, color coded.

Additional escaping of potentially troublesome HTML characters before 
displaying on page for "Contact" field on main page and router detail page.
===============================================================================
}}}
===============================================================================

NOTE: This product includes GeoLite data created by MaxMind, available from 
http://www.maxmind.com/.