[ / / / / / / / / / / / / / ] [ dir / animu / bestemma / britfeel / fast / hisrol / leftpol / sw / vichan ]

/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.
Name
Email
Subject
Comment *
File
Password (Randomized for file and post deletion; you may also set your own.)
* = required field[▶ Show post options & limits]
Confused? See the FAQ.
Embed
(replaces files and can be used instead)
Options

Allowed file types:jpg, jpeg, gif, png, webm, mp4, swf, pdf
Max filesize is 16 MB.
Max image dimensions are 15000 x 15000.
You may upload 5 per post.

New user? Start here ---> http://hydrusnetwork.github.io/hydrus/

Current to-do list has: 1,485 items

Current big job: finishing login and domain managers and overhauling the downloader

YouTube embed. Click thumbnail to play.

6a9f04 No.4550

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v233/Hydrus.Network.233.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v233/Hydrus.Network.233.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v233/Hydrus.Network.233.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v233/Hydrus.Network.233.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v233/Hydrus.Network.233.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v233.tar.gz

I had a good week. The faster duplicate search algorithm is off to a great start, the OS X build is dejanked, and a bunch of bugs (including the recent problem with static gifs) are fixed.

faster dupe search

I went over my old notes for how I wanted to do this and remembered the actual algorithm it isn't as difficult as I thought. I now have a plan for how I want the new fast-search data structure (if you are interested, it is a VPTree) to be stored in the db, and I'm confident it won't be too difficult to get this done.

There is quite a bit of generation and maintenance code to add, but once that is ready I will be able to just swap out the existing slow dupe search algorithm (which needs 200,000 db hits for a db with 200k files) for the new one (which I expect to need something like a couple dozen).

Once we have access to fast dupe search, I will explore adding some sort of 'show me dupes in my db' and 'do this with my dupes' functionality. I'll also eventually be able to add some basic video comparison when I get around to adding a system for animated thumbnails.

I'm ultimately very pleased with this so far.

os x release improvements

The OS X release now looks more like the others. The executable for the client is now 'client', and I've added the server, under 'server'. The app includes the server if you poke around but double-clicking it will still launch the client, just with the new exe, so you don't have to change anything. If you use the extract release, please update your shortcuts!

With the recent support for 'external' dbs (i.e. ones not under the install_dir/db), I would like to migrate the db out of the app release to make the app folder completely read-only (and hence make updating as easy as dropping the new version onto Applications). I need to add some robust db migration code first, but if you use the app, please expect your db to move to your home directory (something like ~/hydrus, but let me know if you have a big preference or there is some OS X standard I should follow) in the nearish future.

A user mentioned they had a permissions/chmod issue with the OS X release, but I couldn't repeat the situation. I have changed up the build process this week, so I might have fixed it by accident. Let me know if you experience otherwise!

full list

- made a plan for faster dupe search

- created skeleton of db tables for faster dupe search

- wrote out search algorithm for faster dupe search when the rest is ready

- updated search algorithm and skeleton to support a future multiframe (i.e. gif/video) similar files comparison

- the os x release now has 'client' as the main client executable

- the os x release now includes the server, under the 'server' executable

- added studio (128,0,0) and meta (0,0,0) default namespace colours

- the 'add all' button on the file lookup tag suggestions panel now will only ever add–it won't remove/rescind pend

- the password system now supports non-ascii input (be careful though, as even a subtle change in keyboard encoding that nonetheless may have the same visual characters will likely be considered a different password)

- fixed the 'nonetype has no dtype' rendering problem introduced by last week's 16-bit channel fix (this mostly affected static gifs)

- fixed non-null ratings changes in the numerical ratings dialog

- fixed culling and adding variables not initialising on the edit html formula panel

- fixed the htmlparser attribute fetcher to deal with both single value attrs ('id') and multiple value attrs ('class'). in the latter case, the many values will be joined up, as how they appear in html

- tweaked disk cache timings a little more to account for more scenarios–it now also reports itself to the shutdown splash screen

next week

I want to push faster dupe search most of all and then likely do some small jobs in the menu overhaul, the panel overhaul, and the new parsing script system.

On a different note, I completed an X-COM: Long War run this past week. I've been playing it exclusively since I think March of this year, and while it was a big time investment and an acutely stressful experience from beginning to end (especially as companion to HAPPENING ERRY DAY YEAR 2016), it is some of the best fun and achievement I've ever had in vidya. If you played the vanilla version of the game and enjoyed it, I highly suggest you check out Long War, even just to watch it. I caught some Ironman Impossible stuff on youtube and got hooked, and then I realised I had to try it myself. I only did a run on Normal, but it was absolutely worth it.

I now return to my backlog, which has grown even more impossible to overcome.

6a9f04 No.4552

>>4550

I remember this was talked about before but now that the portable databases are working nicely how about a command line flag to set a custom window title for the client?


6a9f04 No.4555

File: 4b1b26ef41616a7⋯.jpg (146.97 KB, 1920x1080, 16:9, fb55fbef5a64b673a9318d342e….jpg)

Do we need to worry about the meta: namespace getting some defaultness if we already make use of it? Like is this going to be a new system: type thing I should be shifting existing tags out of to avoid collision with hard coded somethings, or literally JUST some default coloring?


6a9f04 No.4556

>>4555

>meta

It's just a default colour, my good sir.


6a9f04 No.4559

Great work. Animated thumbnails and video de-dupe would be amazingly useful.


6a9f04 No.4560

>>4508

>>4520

Not the only gelbooru subscription that fails lookup, just the most recent. Sometimes they go through, others they don't; not sure if it has to do with any particular search (all of mine are just artist names)

2016/11/24 00:45:17: The subscription Gelbooru Ke-Ta encountered an exception when trying to sync:
2016/11/24 00:45:17: 
2016/11/24 00:45:17: Exception:
2016/11/24 00:45:17: NetworkException: Connection timed out during response read.

  File "threading.py", line 774, in __bootstrap
  File "threading.py", line 801, in __bootstrap_inner
  File "include\HydrusThreading.py", line 157, in run
    self._callable( self._controller )
  File "include\ClientDaemons.py", line 352, in DAEMONSynchroniseSubscriptions
    subscription.Sync()
  File "include\ClientImporting.py", line 2490, in Sync
    HydrusData.PrintException( e )
  File "include\HydrusData.py", line 895, in PrintException
    trace_list = traceback.format_stack()


6a9f04 No.4561

File: b1201321acb9787⋯.png (929 B, 371x77, 53:11, ss (2016-11-24 at 01.28.42….png)

I'm sure these are already suggested, but would love:

Text in front of these ratings, formatted like the tags on the left.

Opening multiple images in the viewer at once, auto arranged in some mosaic way.

Middle/left click the space on the right of the file tabs ("files/files/import/ *here* ") to default open new Files tab.

esc key to cancel manage tags dialogue box, as it does with the other Fkey boxes.

And of course, +1 vote for easier tagging without a new window, tagcloud or most popular etc list.

Thank you for a wonderful software.


6a9f04 No.4562

>>4561

Also, keeping "quick namespaces" saved, so I don't have to keep rewriting the regex out for common imports. Or am I doing something wrong here?

Another vote for optionally bigger thumbnails as well, for those of us with powerful systems that can handle it!


6a9f04 No.4563

>>4550

Just wondering dev, what are your mid-long term plans after the duplicate search rework?


6a9f04 No.4566

> Once we have access to fast dupe search, I will explore adding some sort of 'show me dupes in my db' and 'do this with my dupes' functionality. I'll also eventually be able to add some basic video comparison when I get around to adding a system for animated thumbnails.

really looking forward to that. It's imho one of the most important missing features.


6a9f04 No.4593

File: 415b5ff5640952b⋯.jpg (133.81 KB, 450x638, 225:319, bungaku_shoujo_by_gofu_web….jpg)

Hi Dev, I've found a bug in the "manage options" screen, particularly in the "tags" and "default file system predicates" tabs. Since I have a resolution of 1366x768, I have to scroll down to see all the options. The thing is, the scrolling bar flickers out of existence when it finishes loading, but by resizing the "manage options" window by any amount, the scrolling bar will reappear. There is lag when either of those tabs load, that doesn't occur with the other tabs or even in other menues, and that's when the scrolling bar disappears.

With Love.


6a9f04 No.4597

Hi, I'm getting this error everytime I try to import a parsing script. I've tried all the ones I could find on the board.

http://pastebin.com/JDWa8d91


6a9f04 No.4599

>>4597

Getting this too while trying to create a new parsing script


6a9f04 No.4604

File: 33205d19cd54f71⋯.webm (2.05 MB, 1150x720, 115:72, 33205d19cd54f7127e17524f9….webm)

>>4552

As a db can only run one client at a time, I'll add an option for this to the db!

>>4555

>>4556

Yeah, I noticed this namespace popping up more often, so I just added a new colour for it for new clients. It figured black would be appropriate, but as an existing user you can add what you like!

The client doesn't understand meta at all, yet. Maybe it will, although I suspect breaking current fairly messy meta namespace into different sub groups is probably a better first step.

>>4560

Thanks for this update. My current networking engine is pretty shaky and doesn't recover from many normal errors. I'll make a note of this timeout and see if I can easily catch it and do a retry or two before promoting the error.

>>4561

>>4562

Thank you for these suggestions. I will add them to the to-do list. Showing multiple images in the viewer will have to be a big job, but the others I can put into the normal rotation. Closing manage tags with escape is a bug, so I'll check that this week.

BTW: for now, hovering over a rating will give you the service name as a tooltip.

>>4563

I will put all the big jobs into another poll for people to vote on. I did this at the beginning of this year, taking the top three to get ipfs/suggested tags/faster dupe search, but doing each in turn with all the other distractions and emergencies has taken so long I will shorten the loop and only tackle one thing at a time from now on. The list currently stands at:

(Edit, now I've read the list and remembered: I'll be using my prerogative to do the downloader engine overhaul after dupe search, then I'll jump on whatever is voted on. The downloader engine is in an emergency state, and needs attention as soon as I am free.)

Improve tag censorship, allowing more control over how to hide namespaces from gui/search

Speeding up tagging workflow when inside the media viewer

Displaying files in other ways than thumbnails (like 'details' view in file explorers)

Importing of text and html files

Playing audio for audio files and video files that have it

Support OR search logic

waifu2x interface

random internal db filenames

ability to subsearch import and normal pages, have back/forward buttons like a browser

Selecting thumbnails with a dragged bounding box

Stream file search results in rather than loading them all at once once the whole query is done

Browse mode–popular tags on the left, immediate results on the right

Improve the local booru

Overhaul the downloader engine to use the new parsing system

Client-to-client communications

Improve workflow and pair filtering of tag parents and siblings dialogs

Broaden shortcut customisation

Add more commands to the undo system

Add a repository for sharing ratings

Improve display and zoom quality of very large files in the media viewer

Write a 'tag cloud' control for displaying popular tags in various contexts

Add controls to handle mass adding and editing of subscriptions

Predicate thumbnail border colours on user-editable rating and namespace conditions

Overhaul network engine to allow per-domain bandwidth controls and better proxying

Ugoira support

Supporting cbz (including framework for multi-page format)

Show multiple media at once in the media window, as a mosaic or however else

Have a think about what you would like to vote on, and feel free to suggest anything else.

>>4593

Thank you for this report, I will check this out this week.

>>4597

>>4599

Thank you for these reports, I will check them out this week. Are either of you saving to/loading from directories that have unusual characters? Any Japanese or Cyrillic, or would you say the directories are standard 'C:\users\anon\desktop\blah.png'-like?


6a9f04 No.4605

>>4604

Thanks ! The issue was linked to the path leading to the file. I'm using a few exotic characters such as à é è. Importing a file in a directory that doesn't have any of those characters worked for me.


6a9f04 No.4606

>>4604

Sheeeeit, that's a lot of juicy stuff! I can't wait for the downloader engine personally.

Hydrus has changed my internet life so much that I wouldn't even know what to do without it anymore. It's just so useful and incredible that everything else seems like caveman stuff. The workflow is fast, finding pictures is easy, and the whole user experience is so smooth that I'd never guess this colossal work is done by only one person.

Waiting for a new release every week feels like a religious ritual tbh.

God bless hydrus_dev, keep being awesome!


6a9f04 No.4607

>>4604

Could you consider adding post siblings/parents to that to-do list? It's a feature I adore in every booru.

Thank you for all your hard work


6a9f04 No.4608

>>4604

>Have a think about what you would like to vote on, and feel free to suggest anything else.

I’ve been using hydrus for around two years now, and while I don’t know how you’re handling the complexity of the codebase as more and more features get added, It is something that slightly concerns me. Maybe you could think about compartmentalizing some of the features and sticking them behind an interface so that they’re not as coupled to the hydrus core components. I say this because as hydrus matures, the interface and feature set may start to become quite daunting to new users, and allowing them to pick and choose what features to enable and disable could help their learning of the software. It would also allow users to develop and share modules that they create without having to rely on you to develop them (for example, if there was an interface for taking an image and modifying it while keeping the image’s tags, I would create a module to optimize png files. This same interface could also be used for your planed waifu2x feature).

Although what I’m suggesting is quite demanding and would probably result in one of the largest refactoring you would have to do to the code to accomplish, it might make hydrus more maintainable and usable in the future.


6a9f04 No.4609

File: fceb4d6cb4c4495⋯.jpg (115.19 KB, 850x1060, 85:106, 4eba997aa9a8995554135c0d43….jpg)

hydrus, is the TODO list made public? I'd be interested in seeing all the requests in a single place. Do you keep a list of things that won't/can't be done? I'm also interested in features you yourself are interested in implementing/having/using.

Thank you for keeping the changelog in a single page format like that as well, it's really nice and easy to just see what has changed from version X to Y. I appreciate it.

Take care.


6a9f04 No.4611

>>4604

>Are either of you saving to/loading from directories that have unusual characters? Any Japanese or Cyrillic, or would you say the directories are standard 'C:\users\anon\desktop\blah.png'-like?

Nope, I was getting the error when clicking "fetch response" on a sankaku md5 script I was trying to write. But I tried it again today, with I think the same test md5 and I'm not getting the error anymore, so I'm not sure what happened before.


6a9f04 No.4616

>>4604

I noticed that improving the tag suggestions/auto-tagging using a neural network isn't on the list. Does that mean it won't be on the next poll?

I've seen you talk about it a few times, so I assume it in the plans. And seeing neural networks popping up pretty much everywhere lately I just can't wait until hydrus gets one too.

Also, what does random internal db filenames mean exactly?


6a9f04 No.4620

I have a feature request for the html formula part of a parsing script. I'm having issues with the part that extracts the rating of an image.

Currently it looks something like this:

1st div tag with id = stats

1st ul tag

3rd li tag

The problem with this is that the location of the rating might be different from image to image, even on the same site. Sometimes it is the 3rd li tag, sometimes the 4th. And there is no way to check because the li tags don't have any attributes.

So, can we get the ability to do something like this as well?

1st li tag with contents beginning with "rating"

That would really help.


6a9f04 No.4621

File: 44aae241fa2cb45⋯.jpg (939.85 KB, 2421x3000, 807:1000, 44aae241fa2cb45aa327954fa6….jpg)

>>4605

Great. It should be fixed for tomorrow's release in any case.

>>4606

I am glad you like my program. Your use case is exactly why I do it. I expect and hope to keep putting out weekly releases for a long time.

>>4607

I'm not a heavy booru user, so can you say a bit more about what those relations mean in a booru context, and how that would apply to hydrus?

My understanding is that some files can show a link on their page that essentially says 'this image is a clean/messy/trap/whatever version of this master image'. And are siblings the other alterations of that master?

If this is the case, I would like to introduce some of these concepts in the current round of improving the dupe search algorithm. This will be able to detect small alterations in similar images, and I intend to add a very basic first version of 'dupe metadata', so you can process the detected dupes through a workflow into something useful. A row of dupe metadata would be 'this image is (relation_type) to this image', where the relation could be 'a lower quality jpeg' or 'a smaller version of', or indeed 'a child of'. I can use that metadata to merge tags across dupes or load it with an image and display the file's parent/siblings/whatever as links from the media viewer, for instance. Is that what you are looking for?

Can a child/parent/sibling be a very different image, like different pages in a comic, or are they always very similar in appearance?

>>4608

I agree. Beyond a hundred different general cleaning jobs (cleaning up my old crap code into something I can modularise easier), my push towards the new parsing engine is intended to be a step in this direction–I no longer write the scripts or downloaders, I write the engine that runs them. I'd love if the program were very modular in future, with browser-like plugns and so on. I am completely not opposed, but it will as you say be a lot of work.

The complexity of the codebase is actually less than previous, as I've cleaned up some colossally bad code. Still a lot to do, but it isn't a big concern on my end yet.

The idea of switching features off is also good. The new tag suggestions stuff is all turned off by default, and pushing more of the complicated stuff this way is a good idea. Maybe I could add 'simple/advanced' mode to the help or something, to hide tag siblings and other complicated things.

>>4609

It is all on a ToDoList (AbstractSpoon) on one of my monitors, so I can't share it easily. Just 500-odd things like '.txt export needs tag service discrimination'. I don't have any can't/won't stuff, I don't think. People sometimes suggest technically unfeasible things, in which case I try to politely say "you'll probably have to wait ten years for python to get better". I appreciate that other people think in very different ways, so I'm trying not to force any sort of workflow on someone who would prefer to do something another way. I suppose I'm probably resistant to any normie-fication of the program, but other than that, I don't think I have any triggers.

There's a lot of stuff I don't know how to do, but I just read up on it and muck about with a library and then I'm off. I enjoy doing a bit of everything, rather than specialising.

For me, I'm interested in the big stuff, not so much because I particularly want it in my own client but because I enjoy working on and completing difficult things. Faster dupe search, and the downloader-scripting overhaul. And neural net tagging is something I really want to learn about, as I know nothing about how that stuff works yet and I reckon it could be a gamechanger.

>>4611

Let me know if you get it again, and if you do, send me the traceback (in fact, it is probably in your client.log, in your db dir). It might be a similar looking error that is coming from a different part of the code.

>>4616

Thanks! I think we first talked about that after the last poll, and I forgot to add it. I'll add it to the list.

A user asked for an option for another layer of obfuscation in the file system by scrambling the filenames, which would not be a simple job, so I added it to the list.

>>4620

I'll make a note to add a test like this, thank you.


6a9f04 No.4626

File: e71f8ba6b675217⋯.png (61.34 KB, 760x256, 95:32, local_booru_html.png)

>>4621

danbooru has a nice definition of what a post relationship is, plus when they should, or should not, be used:

http://danbooru.donmai.us/wiki_pages/21857

It says "Parent and Child Posts, also known as Post Relationships are a means of keeping similar looking posts together", and that includes different resolutions, crops, derived works, vector traces, different costumes, etc. Maybe even static images that came from a gif or a shorter segment of said gif. I attach an image that illustrates this perfectly. The link I provided also contains links to examples of these cases.

I believe the dupe search algorithm will be very helpful to find related images, but in cases of mirrored images, with transparency or even parodies (like the image I attached) might be a little bit too much for any dupe search engine.

Finally, boorus advise against using this file relationship for sequences of images or images that are very different from each other (like that of a comic). For these cases, boorus suggest using "pools":

danbooru.donmai.us/wiki_pages/43039

Pools are basically kind of like the "collect" option in Hydrus, except they are manually created, you can assign them a name and set the order of the files. It's less messy than tagging a bunch of files with page:00001, for example.


6a9f04 No.4631

>>4621

>Let me know if you get it again, and if you do, send me the traceback


'charmap' codec can't decode byte 0x90 in position 17378: character maps to <undefined>

Traceback (most recent call last):
  File "include\ClientGUIParsing.py", line 1409, in EventFetchData
    self._example_data.SetValue( example_data )
  File "site-packages\wx-3.0-msw\wx\_core.py", line 13075, in SetValue
  File "encodings\cp1252.py", line 15, in decode
UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 17378: character maps to <undefined>


6a9f04 No.4661

File: 27c794a8367851a⋯.jpg (701.64 KB, 1228x760, 307:190, 27c794a8367851ad86f75ef4c5….jpg)

>>4626

Thank you for this information. I would like the new dupe metadata to support exactly this kind of relationship, and the better dupe search will help in finding most of these cases. Matching crops and animation stills will have to human-initialed, but I won't block them at all.

Let me know what you think as I roll this stuff out. I don't think I can support everything in this first version, but I think we can move it a little forward.

>>4631

Thank you for this. This is very odd–it looks like the server you were requesting from returned some binary data or something. I will make a note to add some better error handling here. Do you remember the site and general query you were sending?


6a9f04 No.4664

File: 589ff2e5b41b144⋯.png (2.1 KB, 512x84, 128:21, temp.png)

>>4661

>Do you remember the site and general query you were sending?

Use this parsing script, and test with this md5: eea8d884f3127c7a4024c531e4c1f23e


6a9f04 No.4665

Please add a way to add multiple parsing scripts to a list which will execute on a specific image one after another until a match is found.

On the "option>tags>file lookup scrips", have a list where you can add scripts from your main list of parsing scripts, and move them up and down in the queue. Then add an "auto" button on "manage files's tags>file lookup scripts" which runs all of the scrips in the list until a match is found.

This would really be helpful.


6a9f04 No.4685

>>4664

Thank you for this. I have fixed the problem for tomorrow's release. The html downloads correctly and the tags parse as well.

>>4665

Thank you, this is a good idea. I will add it to my todo.



[Return][Go to top][Catalog][Nerve Center][Cancer][Post a Reply]
Delete Post [ ]
[]
[ / / / / / / / / / / / / / ] [ dir / animu / bestemma / britfeel / fast / hisrol / leftpol / sw / vichan ]