V550-Reference

From Newsbin

Reference Page for Navigation

This Reference page allows you to navigate to help on all Features of NewsBin. The picture of NewsBin above has all possible tabs showing, allowing you to click on the picture to navigate to the Help for each feature. Other images with this icon work the same way.

Title Bar

This is at the top of NewsBin. It displays the configuration file name, version, and build number. In this case, the configuration file is NewsBin.nbi.

Menu Bar

If you are an unregistered user, the Menu Bar looks like this:

Registered users have the "Register Newsbin..." option removed:

Detailed help for each menu option are available at the following links:

Toolbars

Newsbin has three toolbar styles, Big icons, Small icons, and the default, Buttons Only. Choose which one you prefer on the View menu option. They all represents some commonly used functions.

Big Toolbar

Small Toolbar

Just smaller icons to save space on the screen.

Button Toolbar

Saves some space and clearly labeled for people who prefer words over pictures.

Tabs

NewsBin has a "Tabbed" interface to separate out functional areas.


Groups	Search	Servers	AutoPAR	Thumbnails	Image Database	Post List
Download List	PAR List	Failed Files	Files	Logging	Connections

Status Bar

Some useful runtime information is displayed and updated in the Status bar at the bottom of the NewsBin screen.

Options

Newsbin has a wide variety of Options you can set to customise its behavior. The default values are appropriate for most users as they start using Newsbin, but as your use of Newsbin grows you are likely to want to customsie some of the settings to suit your own particular manner of working.

Critical Error Messages

* As of version 5.56, Critical Error Messages are displayed in a Popup Window. 
This popup window can optionally be disabled by unchecking Options->Confirmation Windows->Error Popup,
or you can use the Hide till Restart button.

Inner Workings

The following sections discuss details of how Newsbin handles certain things internally, so the user can have a better understanding of why things work the way they do.

Header Storage

Headers are stored in the SPOOL_V1 folder located in the Newsbin Data folder. Under SPOOL_V1, there is a folder for each group, and within each group folder, there is a folder for each day of headers retrieved from all servers. These folder names are based on the number of days since 1970. So, if today is March 12th, 2009, the folder name would be 14316 and yesterday was 14315. Within each of these folders, the actual headers are stored in .RV3 files, named after the hours of the day.

When you do a header update on a group, Newsbin checks the range.db3 file in the SPOOL_V1 folder to get the article number of the last header it retrieved, for each server if you have multiple servers configured. Newsbin then performs an NNTP XOVER command asking for all headers available since the last header was received. If you have a Header Overlap value set in the news server configuration, Newsbin will use that value and subtract that from the beginning article number to pick up any headers that may have shown up due to server backfilling. By default, this value is set to 0.

There are several ways that Newsbin loads headers into a post list for viewing. The two basic methods are using Show Posts and Show All Posts. If you use Show All Posts, Newsbin blindly loads all the spool files for that group, regardless of age. For many groups this can take a good deal of time and memory.

The Show Posts command is based on the Display Age setting under Options/Setup. If you have a display age of 14 for example, Newsbin will say "what's the current day" and if your PC's date is correct, it'll say today is 14316 (assuming March 12, 2009). Then it subtracts 14 days from that and loads 14316, 14315 ... 14302. There is some overlap so, you might get 14301 too. If the date and time of your PC is wrong, you may get unexpected results, including no posts loaded.

If the logging level is set to "Debug", you will see Newsbin loading each RV3 file in the Logging tab. You can also look in the folders at the RV3's after an update and see which ones got updates.

RV3 File Format

All the headers from all the servers are combined into a single RV3. They're split up only by date and group. Duplicates are removed when detected. Each record represents one combined file.

typedef struct   _xoverhdr_v4 
  { 
     uint32_t   m_nVersion; 
     uint32_t   m_nRecordLength; 
     uint32_t   m_nUnixDate; 
     uint32_t   m_nPartSize; 
     uint32_t   m_nFlags; 
     uint16_t   m_nPartMax; 
     uint16_t   m_nCrosspost; 
     uint16_t   m_nParts; 
  }XOVERHDR_V4;

Every record starts with one of these. The m_nRecordLength is the length of the entire packet including this header. Following this headers is the:

Subject From Group

Variable length, null terminated strings.

These are followed by 1 or more

Then the next packet starts.

RV4 File Format

As of version 5.54, RV3 stored header files are automatically upgraded to RV4 which are combined on a per day basis in order to lower total file count. Reducing the number of files accessed should help improve performance.

A one time conversion of any existing RV3 files are performed when you do a Show Posts or Show All Posts function (loading headers from disk).

Searching Spool Files

From the Search tab, if you have it set to "Local" mode, Newsbin will search headers stored on disk. The first time you perform a local search, Newsbin catalogs information from the .RV3 files into a Free Text Search database, FTS.db3. There is one of these databases in each group folder under the SPOOL_V1 folder. The first search can be very time consuming while this database is updated. Subsequent searches will be very fast.

Duplicate Detection

The Duplicate Detection is the mechanism Newsbin uses to determine if you have previously downloaded a specific file before. It is used for Duplicate detection and a "Signature" of each file you download is saved in a database called signature.db3. If you turn this option off, Newsbin will not check for duplicate files being downloaded. This option is on by default. If you want to force the download of a duplicate file while keeping the signature cache active then use the "Add to Download List - Bypass filters" option (keyboard shortcut of CTRL-Y).

Geek Details

Newsbin performs a calculation across the first 24K of each file downloaded to create a "Signature" which is compared against stored signatures from other file downloads in the signature database. The NNTP protocol has no provision for stopping a download short of killing the connection. Since many news servers do not acknowledge a lost connection as fast as Newsbin can create a new connection, you could easily go over your server's maximum simultaneous connection limit and start to get rejected by the news server if Newsbin immediately killed the connection. In order to avoid this, Newsbin continues to download the rest of the post and throws it away. With today's high speed connections, it isn't as much of an issue as when most people were on dialup.

Download List Database

The download list and scratch list is stored in downloads.db3

With the db3 file, the list now saves as files are added, removed, downloaded, or failed.

Updates are written to a downloads.db3.journal as needed so the entire download list is not saved all at once as previous Newsbin versions had done with Autosave.nzz files.

This allows for much faster time to Exit and completely unload Newsbin from memory.

AutoPAR Database

The file "Autopar2.db3" contains tracking information about the currently downloaded PAR2 files and the state of the current downloads. If Autopar stops working, this file might be corrupt. Exiting Newsbin, then moving the file out of the data folder, is a valid way to fix a corrupt Autopar2.db3 file. When you restart Newsbin will re-create the file.

Image Database

The Image database can be enabled in the switches options. By default JPG/PNG and GIF files will be added to the database along with the group, subject and from field. The image will also be left on disk. There is a hidden NBI option to disable saving the image to disk. The databases are "per group" so, each group's images are separated. Using the "Image Database" tab, you can browse all of your old images including searching and sorting the images. The images can be exported from the Image Database tab in either raw image format (JPG/PNG/GIF) or as CBZ files. CBZ files are essentially combinations of image files into a single ZIP file which can be read by "comic book reader" image viewers like "cdisplayex".

Reading Text Posts

Text posts are stored in the "Read Posts" folder in the Newsbin Data Folder as .txt files using the subject of the post as the filename. Download text posts by using ctrl-r or right-click the post and select "Read Post Body".