windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v245a/Hydrus.Network.245a.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v245a/Hydrus.Network.245a.-.Windows.-.Installer.exe
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v245a/Hydrus.Network.245a.-.OS.X.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v245a/Hydrus.Network.245a.-.OS.X.-.Extract.only.tar.gz
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v245a/Hydrus.Network.245a.-.Linux.-.Executable.tar.gz
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v245a.tar.gz
I had a very busy three weeks overhauling the network and much else besides.
This release increments the network version. v244 and earlier clients will not be able to talk to v245 and later servers and vice versa.
network overhaul
This took two weeks longer than I wanted it to, but it probably had six times as much work as I expected. Every old creaky part of the network I improved revealed a critical link that was A) just as bad and B) had to be updated or else nothing would work. Github reckons I made 14,380 additions and 10,911 deletions. A typical single week has 600/300, which shows at the very least that I typed a lot more than usual. I ended up putting in a lot of hours and exhausting myself for v245, but I ultimately had fun, and I am really happy with the overall result.
The technical upshot is that the network runs better, faster, and uses far less bandwidth. The repository update process–which is the overwhelming majority of all network traffic–now uses about 23% of what it used to, and it runs on a new philosophy that allows much better scaling in future.
Repository processing now occurs in a more sane way and inside single database transaction, meaning it will not pause to save every 100,000 rows. Processing the entire PTR should now only take 1-2 hours on a typical PC.
'Services' objects are also completely rewritten, and review services and manage services along with them. These panels let you do more and show more information–you can now pause repositories and also force processing from review services.
Bandwidth tracking and enforcement is improved as well. Services that connect to the internet and accounts for repositories now know how much data they use and support multiple flexible rules for when to stop doing heavy lifting. These new bandwidth controls will soon appear in more places across the program.
If you are a server administrator, you now have these bandwidth controls for your services and account types–please check them after updating to make sure they are set how you like.
unfortunately a full resync is required
Unfortunately, the changes are so significant that the old update files cannot be converted. Your clients must resynchronise with their repositories. For my public tag repository, this is now about 620MB of data (down from 2.8GB!). I have set a 250MB/month and 50MB/day limit on existing repositories just to smooth out the hump of all of us resynchronising. Please feel free to remove these restrictions under manage services when things calm down.
If you are a pro user and sync with the PTR, please download this: http://www.mediafire.com/file/1s1ee30fk27ejc9/ptr_update_files.7z , extract it somewhere, and then import its contents via the new menu entry services->import update files. This skips you having to download the files through the network and saves me a bit of bandwidth. Thank you!
some things don't work yet
Despite all the work, there is still some more to do. Review services and manage services still have some placeholder panels, and I haven't had time to fix/update some database maintenance and network administrative functions. Please hang in there as I reintroduce these things over the next few weeks.
I tested this extensively by myself and then tested it again with users. I have cleaned out many bugs, but there are likely others. Please report them and I will turn them around as quickly as I can.
full list (you do not have to read all this)
- fixed a v244 update problem for clients updating from <v238
- some misc stuff:
- if you start editing many subscriptions, cancelling a single dialog will break the chain of loading new dialogs
- reduced some redundancy in regular client file import
- improved how the dialog for selection from a list of strings works
- if the client ever merges one directory to another (such as in external file locations migration) and any files fail to merge, the source will no longer be deleted
- created new flexible bandwidth tracking and ruling objects
- updated how repository updates work, splitting the old explicit and self-contained content update package system to a new implicit definitions/content split system and an improved one-step-sync metadata propagation
- updating now takes approximately 23% as much bandwidth as before
- update files are now stored in client_files and server_files like any other file (client_updates and server_updates folders will be deleted on update)
- the server will print update generation info to its log
- unfortunately, updates cannot be converted, so a complete resync of update files is required. the smaller update size and better bandwidth controls should mitigate the problem somewhat
- the server has been compacted across all content types–its mappings db file should shrink about 12%
- due to new service-specific ids, server.master.db should increase in size, typically about 50%
- harmonised how GET and POST/response args are built and parsed across the network
- server administration initialisation is now simpler, done with 'init' registration key
- leading and trailing spaces are now removed from both the namespace and subtag components of namespaced tags, meaning 'title: blah' will be collapsed to 'title:blah'. tag repositories will clean their existing tags on update
- invalid serverside tags will be replaced with valid placeholders on server update
- refactored, harmonised and simplified some server request code
- Eris makes a triumphant return to the root welcome page ('/' request of any service) with improved self-description text as well
- updated how bandwidth is tracked and overseen
- all requests now consume bandwidth
- fewer requests are actually constrained by bandwidth–at the moment, it is only update files and file/thumbnail downloads, as these represent the overwhelming majority of bandwidth consumption and are not at all critical to service operation–this may change in the future, but it suits our purposes for now
- bandwidth tracking code is more sane across the board
- the server's administration service now tracks server-wide bandwidth as it happens
- service and server-wide bandwidth rules are consulted as soon as the request begins
- improved some server response rendering
- permissions are more flexible and content-specific
- improved how file repositories check and process file requests
- created a gui control for managing bandwidth rules
- updated the serverside service object
- the serverside service object now contains bandwidth rules and tracking
- updated how the server deals with services on a db level
- refactored and cleaned a ton of server db code
- updated how services are edited over the network
- updated how the server associates its services with its http pipeline
- converted clientside server service management gui to the new panel system
- updated gui code for clientside server service management
- converted clientside server service management db code to the new system
- improved how bandwidth errors are reported
- updated the account type object
- the account type object now contains bandwidth rules and a more flexible permissions system
- updated how the server deals with account types on a db level
- converted clientside server account type management gui to the new panel system
- updated gui code for clientside server account type management
- serverside account types are now fetched from a cache, reducing memory sprawl
- updated the account object
- the account object now contains a bandwidth tracker
- updated how the server deals with accounts on a db level
- updated the clientside server object
- the clientside server object now contains bandwidth rules and a tracker and has improved error management, recovery, and reporting
- updated how the client deals with services on a db level
- improved how serverside bandwidth errors are caught
- all clientside services start with a 50MB/day bandwidth limit
- existing repositories will get a 250MB/month, 50MB/day default bandwidth limit, just to help us get over the hump–see the release post for info on how to get all the updates anyway
- updated manage services dialog extensively
- service account registration now occurs through a simple button on the normal clientside service edit panel
- all services can now be renamed
- updated the content processing pipeline
- prepared code for future merging of file and tag repositories
- added future support for pend-petitions for files and mappings and simple creation permissions for tag siblings and parents
- the clientside content processing pipeline now operates inside a single database transaction, reducing a great deal of previously redundant hard drive activity
- the serverside content processing pipeline now creates per-service timestamped content definitions
- the clientside content processing pipeline now maintains and appropriately consults a cache for server definitions
- pre-process disk cache is more intelligent
- updated review services to match the new service objects
- service review panels now show more information about error state and so on
- service review panels should now update as soon as services change
- you can now force sync repositories from review services
- you can now pause/resume repositories from review services
- you can now export repository update files from review services
- you can now import repository update files from the services menu
- repository thumbnail download now uses a cache for faster thumbnail ownership testing
- culled a lot of old code and experiments
- deleted all the old messaging depot code and db table cruft
- removed optimised petition processing
- serverside deleteorphans is temporarily disabled
- hydrus client-repository relationship no longer supports news
- removed the 'stats' admin service call–it will come back as a account review page
- clientside clear orphans is temporarily disabled
- clientside local file/thumbnail server is disabled for now
- custom service 'messages' for the root page are no longer available
- some things are not working yet–they will be back in soon
- content presentation on review services
- service reset, other advanced content service controls
- ipfs controls on review services
- local booru controls on review services
- most service-specific panels on manage services
- petition resolution
- serverside account modification, including banning
next week
I now have three priorities: catching up with the urgent bugs that have piled up in the past three weeks, finishing off the placeholders of this overhaul, and resuming duplicates. I'd like to fit in a little sleep as well.