/hydrus/ - Hydrus Network

Can we get parsing for reddit and downloading from gfycat plis?

>I am also considering just adding a rule to convert all underscores in tags to spaces.

be careful with this, many artists have underscores in their professional name and that would fuck those tags up.

Some artists with underscores in their names have characters named something super common like "joe", so people use the tag character:joe (artist_name) to clear out which joe is in the picture.

here's my suggestion: apply this rule only to non-namespaced tags.

also, do you have a list of sites/imageboards to make parsers for?

if you do, here's a few to add

furaffinity.net

inkbunny.net

weasyl.com

lulz.net

u18chan.com

yeah yeah i know, im a huge degenerate please rape my face

>>8101

On the underscore

Is it hard to do or time consuming?

Does the work need to be done to advance anything major or is it cleanup work?

If its something that needs to be done to implement something else, by all means do it, I honestly could not give one flying fuck about the removal as there is little use to keeping them

However if it takes real time you could devote elsewhere, and is not important for a new feature implementation, then fuck it, put it off till you have an off week where you aren't feeling up to writing code and cant really do anything major but still want to accomplish something.

>>8105

how do you create sorting schemes by custom tags?

update on this shit

>>8111

>>8110

it seems to have fixed itself, after i went into services and refreshed the accounts.

Will the new parser be able to find hashes from URLs and check if the file is already in database before (re)downloading it?

For example: blah.blah/file_store/830914cdf228e2dd713a2dc3e8df195ce17cd649.jpg &

blah.blah/_images/830914cdf228e2dd713a2dc3e8df195ce17cd649/somefile.jpg

I haven't had a chance to mess around with it yet.

>>8116

Go to "file->options" then go to "sort/collect" then type the namespaces you want to sort or collect by, separated by - characters

namespace1-namespace2-namespace3

Because - is the separator you can't sort or collect by any namespace containing a - character which is a weird limitation but applies nonetheless

>>8125

Thanks fam. Seems like a simple workaround would be for the separator to be changed to double – though im not sure just how many namespaces exist with hypens already in them

Hey fam retard here, if I want to blacklist a tag like say 'black hair' how do? I don't understand how this works

>>8128

black list? what do you mean by this? you mean use it as a search parameter?

if thats the case, just type it with a - at the beginning, ie:

anime

-black hair

it will bring up all pictures tagged anime and rule out any that are tagged "black hair"

If you force Hydrus to process PTR updates with the "process now" button, and then stop it after a while, the client will freeze for a long time after the "processing updates finished" popup. On a smaller db it took about 15 minutes and on my main larger db I left it for 2 hours before I ended the process.

Any way you can make it not freeze like that? Thanks

>>8128

Use the censor function. Right-click a tag.

There is no good answer for spaces vs underscores. The internet has chosen not to conform to a single standard.

Copied from a stackoverflow answer you potentially have to deal with

text with spaces

textwithoutspaces

encoded%20spaces%20in%20URL

underscore_means_space

dash-means-space

plus+means+space

camelCase

PascalCase

" quoted text with spaces" (and single quote vs. double quote)

slash/means/space

dot.means.space

>>8130

I also observed this. I'm going to guess it's sqlite doing its thing…?

>>8132

Is right.

That said, I feel regardless that just getting rid of underscores by default is probably a good idea.

The problem should lessen anyhow if eventually more better downloaders pre-sanitize tags from more sources.

>>8136

>all the windows fucked up and shit

>that scared "w-what happened?" crying face

I say you leave it for the novelty.

>>8139

Haven't see the massive gifs but that would be funny to see, pretty much everything massive of mine is massive for a reason, a good reason? no, but a reason.

when you mention decompression bombs and treating them as such, there is an issue, what if I want the image/video? I have to go around turn the setting off and then back on.

Honestly, with both of these... well in a download page you get a log, and it will tell you failures and you are able to retry the failures.

Is there a way to implement a right click menu to 'override filter'

This way it can force import a file that it either sees as a decompression bomb or as just a massive fuck off file size.

I think when implemented I want to set it so everything over 20mb is a check with me moment, im looking at files and seeing where the threshold is for file size, 20 returns 600~ files, 15 returns 1200~

I think 8mb may be good, would have to test it a bit but 8 returns 5000 files and given how much I have downloaded, its still next to dickall and I would be able to manually ok them, however if I go that low an ignore filter right click menu option would be needed.

On that note again, would it be possible to add another type to what the program tells us?

we got successful, already in db, deleted, and failed

would it be possible to change failed for filter reasons over to filtered? this would be decompression bombs and this maximum file size, just so its brought to are attention that something needs checking better then failed.

that also said, can we get an 'ignore filters' option to manual imports? this would make it easier to import things like decompression bombs and a max file size without disabling it in options each time.

Thank you for voting in the poll! There are many more 'please make it optional' results than I expected, so I am glad I ran it. I will make this change more slowly and have it as a more flexible and optional system.

>>8103

I'm from a low-tech background of running ftp servers under desks, so that's how I run the PTR. It has always just been an old computer I have sitting in a corner, much like how server.exe is a thing you can just throw on a USB stick if you want.

I don't know much at all about VM stuff or more advanced hosting solutions. I wonder if hydrus will ever be a big enough network to go in that direction, or if I will eventually move the server's jobs down into the client and have more and more of a P2P system.

If you know about that stuff and we ever run into a serious problem here, I'd be interested to know what you think would be an appropriate solution.

>>8105

Thanks. I will write an exception or something into my new thumbnail banner generation code to detect integer namespaces and deal with that like "min-max", like before.

>>8107

I am currently working on a big overhaul of my downloader engine. Once I am done with the current parsing help, I will move toward converting the gallery downloaders to the new system. At this point, it will be much easier for me or any other user to create parsers for lots of new sites. Let me know how it works for you as I roll it out!

>>8108

Thanks, this is a great idea.

My general thinking has been that the work of fixing underscores where they are actually desired (by substituting the double-underscore escape character) would be an easier job than the current mess of fixing every instance of 'blue_eyes', but yeah, perhaps a more granular system is the key. Most of this shit is in unnamespaced tags.

In any case, since people do care about this after all, I'll have a more complicated system than just 'replace them all at the db level wew' anyway.

>>8109

Thanks. I will add these to the list. I don't want to get super-bogged down doing this myself (I made the new system user-editable and -sharable precisely so other users could take this work off me), but I will absolutely write these down. If you are HTML or JSON fluent, you might like to get involved in writing some parsers yourself as these systems roll out. It looks like some of the work is going to happen on discord and github–I'll link it all everywhere as this becomes real.

>>8110

>>8111

>>8117

Thank you for this report. Sometimes the client has trouble getting a new session key from a repository. I think there is a timing issue. It is now all working on the new login manager, which at the moment is basically two sticks and a short bit of string. The best solution at the moment is to give it a couple of days to figure itself out or yeah just hit refresh in review services. I'll deal with this properly and improve error messages when I get the login manager proper going.

>>8115

Thank you for your input. I agree personally, but in making hydrus, I have found the entire concept of 'tag siblings' and generally 'A should look like B, imo' is the most complicated and divisive part of the whole deal. People's opinions on whether one style is better or worse vary wildly and for valid (albeit subjective) reasons. I tend to not care all that much about how tags look, but others spend hours on it.

I expect to put more time into several new systems for tag presentation in future, and iterations on existing tag siblings/parents to allow more user-side choice. It does take a lot of time and work, unfortunately.

>>8121

Thank you for this follow-up. I think that 'loop=1' business is the page of images downloader getting the 8chan 'webm viewer' page, like so:

https://media.8ch.net/file_store/e95d2a111b5d260016b9128bb3d1dc6ee178328f3d5d8589b3a403bb21dccaa0.webm

The 'uninteresting mime' part is due to that link actually returning an html page. It looks like it is doing the images ok, since the thumbnail link goes straight to the raw image. What a mess.

Future versions of 'page of images' (when I move it to the new parsing system) will be more flexible and allow fixes to this in the downloader page ui itself, but it will be a while until that is out.

I do not know a clever solution to this at the moment. This is a legacy issue with 8chan that affects a finite number of years-old files, so I don't want to invest a bunch of time fixing it in the thread watcher proper. I will make a fix, though–but let me think about it a bit. Maybe I can bump the new 'page of images' up.

>>8124

Yes! And in a generalised way: If you can figure out a way to find a hash in the HTML or JSON of the target page, you can now decode it from hex or base64 (to raw bytes) and then inform the new parsing system whether it is md5, sha1, sha256, or sha512. The client will check and decide whether to download (or just set 'already in db'/'previously deleted') appropriately.

>>8125

>>8126

This ui is years old and sucks. My apologies–it'll one day be neater.

>>8130

>>8133

Yeah, unfortunately sqlite takes a while to clean up big transactions. We are doing big server-level stuff on desktops, so it can be tricky to schedule this stuff in a way that fits nicely with an ui. If the db writes a million new tags, that's a lot of 4KB blocks to update from the db-journal to the main db file. During that time, the ui won't work well. I usually hide this big delay behind idle time or shutdown maintenance time where you never notice, but if you hit process now, you have to deal with it on its face.

I have updated the warning dialog on 'process now' to say it may take a while to give your ui back even after a cancel. But I don't want to put time into making this too neat, as it has always been more of a debug thing. I recommend that all users let the client do update processing according to the regular maintenance rules as work for them under options->maintenance and processing.

Having said that, 2 hours sounds way too long–is that drive especially full or fragged? If you are running on a HDD that is also a bit slow for other reasons, I recommend only doing update processing during shutdown time. Limit it to ten minute bursts or so and you'll never get massively boshed. Or pause/slow PTR syncing until you next update your hardware.

Although, it could also be that in the long time it took to clean up the old transaction, let's say 20 minutes, idle mode kicked in again and the whole thing restarted! I feel like I have seen this before–again, I recommend not using the 'process now' button except for debug or other testing purposes. I didn't design the system to generally be used pleasantly in that way and didn't hide it behind help->advanced mode for no reason.

>>8132

>>8133

Yeah, my general feeling is that this is an endless battle. There is no ultimate perfect solution, so all I can really work on is a system that lets people head towards what they would prefer.

My fingers-crossed hope is that neural nets will make this all a lot easier to manage in the coming years.

>>8136

Thank you. Please try hitting help->debug->report modes->hover window report mode and then play with these broken hover windows in a variety of ways. This will spawn a fucking ton of popup messages.

Then shut down the client, go into your log, and find all the popup stuff, bundle it into a new text file or something and pastebin/email/discord DM it me! And observations you have about when the popups came in most of all (e.g. 'only came up on initial load' or 'got lots during flickering while mouse over the area they should have been') or whatever would also be helpful.

>>8143

Thanks. I've decided to add this new 'kill big gifs from THE SFM COMMUNITY' rule to 'file import options'. Now I am thinking of it, it would be reasonable to move the 'don't import decomp bombs' to file import options as well–then the retry action would only be another three clicks–'open file import options', 'decomp bombs are ok', 'apply'–rather than the pain of going into the full options.

After taking more feedback from the discord yesterday, I am zoning in on 32MB being the default for big gifs. It'll be editable if people want different, but the main thing I want to cut out is the stealthy 168MB gif that serves no honest purpose for anyone.

I will keep the ideas of adding a new file result status and having some sort of 'override filter' pipeline in mind, but these would probably be quite complicated to actually do well, so I'll try it simply through file import options first and we'll see how that goes.

If you like some decomp bombs, I recommend you just turn that option off. I've never encountered a malicious file in the wild, and the bad pngs we are talking about don't usually do any real harm–they just blat your CPU and memory for a handful of seconds.

>>8144

When I hover over where the window should be in the viewer it will appear and display correctly without flickering but just in the wrong position.

But if where the window appears and where it should be overlap and you mouse over it, it will flicker in and out repeatedly.

During the flickering a bunch of message pop-ups will appear, they happen each time it fades in or fades out.

Also if I move the media viewer window between different workspaces, the hover-over windows will (sometimes) be in another random position.

Here is the log: https://ghostbin.com/paste/xqhmc

Also separately I've gotten these exceptions sometimes when using the search bar:

2018/02/25 21:43:52: Uncaught exception:
2018/02/25 21:43:52: AttributeError
'AutoCompleteDropdownTagsRead' object has no attribute '_dropdown_hidden'
  File "/opt/hydrus/include/ClientGUI.py", line 3442, in TIMEREventPageUpdate
    page.TIMERPageUpdate()
  File "/opt/hydrus/include/ClientGUIPages.py", line 771, in TIMERPageUpdate
    self._management_panel.TIMERPageUpdate()
  File "/opt/hydrus/include/ClientGUIManagement.py", line 3345, in TIMERPageUpdate
    self._UpdateCancelButton()
  File "/opt/hydrus/include/ClientGUIManagement.py", line 3202, in _UpdateCancelButton
    self._searchbox.ForceSizeCalcNow()
  File "/opt/hydrus/include/ClientGUIACDropdown.py", line 583, in ForceSizeCalcNow
    self.DropdownHideShow()
  File "/opt/hydrus/include/ClientGUIACDropdown.py", line 383, in DropdownHideShow
    self._ShowDropdown()
  File "/opt/hydrus/include/ClientGUIACDropdown.py", line 322, in _ShowDropdown
    if self._dropdown_hidden:

2018/02/25 21:43:56: Uncaught exception:
2018/02/25 21:43:56: AttributeError
'Panel' object has no attribute 'IsActive'
  File "/opt/hydrus/include/ClientGUI.py", line 3442, in TIMEREventPageUpdate
    page.TIMERPageUpdate()
  File "/opt/hydrus/include/ClientGUIPages.py", line 771, in TIMERPageUpdate
    self._management_panel.TIMERPageUpdate()
  File "/opt/hydrus/include/ClientGUIManagement.py", line 3345, in TIMERPageUpdate
    self._UpdateCancelButton()
  File "/opt/hydrus/include/ClientGUIManagement.py", line 3202, in _UpdateCancelButton
    self._searchbox.ForceSizeCalcNow()
  File "/opt/hydrus/include/ClientGUIACDropdown.py", line 583, in ForceSizeCalcNow
    self.DropdownHideShow()
  File "/opt/hydrus/include/ClientGUIACDropdown.py", line 381, in DropdownHideShow
    if self._ShouldShow():
  File "/opt/hydrus/include/ClientGUIACDropdown.py", line 266, in _ShouldShow
    tlp_active = self.GetTopLevelParent().IsActive() or self._dropdown_window.IsActive()

>>8144

> If you are HTML or JSON fluent, you might like to get involved in writing some parsers yourself as these systems roll out.

i'd love to but im a fucking brainlet who cannot learn how to do things unless he's taught by a person. i've tried many times and i just lose focus and get frustrated when shit doesn't compile or math gives an incorrect result.

i'll just wait and give you support by keeping tabs on errors i get.

in other news, i've tagged at least a couple thousand pictures already out of those sites i mentioned earlier, the battle never ends.

>>8144

>Having said that, 2 hours sounds way too long–is that drive especially full or fragged? If you are running on a HDD that is also a bit slow for other reasons, I recommend only doing update processing during shutdown time. Limit it to ten minute bursts or so and you'll never get massively boshed. Or pause/slow PTR syncing until you next update your hardware.

It's a db with 70k files. client.mappings.db is 5.55GB, client.master.db is 1.66GB. It's on an AES encrypted 1TB 5400rpm WD green HDD. Not full or very fragged. So yeah, not exactly super fast. Works fast enough for anything else in Hydrus though.

>>8144

Since today morning the client doesn't perform any subscription checks or downloads.

I can do duplicates, maintenance and search for files. But it doesn't check nor download.

bandwidth rules are ramped up, so there should be no problem.

Multiple subscriptions don't want to check/download, furthermore newly added subscriptions don't check/download as well.

It simply stopped in the middle of downloading in the morning and after restart never continued to download.

Note: I could only kill -9 it, because it didn't close even after an hour

>>8149

Welp, false alarm. Somehow paused the subscription syncronisation.

Some hotkey or however.

>>8149

Same thing happened to me a while ago and I haven't been able to get it to work again so far.

They are all stuck in a state that says they will check again when after I press OK to exit the subscriptions window, but they never start to check.

>>8144

>Yeah, unfortunately sqlite takes a while to clean up big transactions

I also think it has pretty bad HDD access patterns when it can't run fully in memory. At least I always had big performance issues with bigger sqlite databases when things like postrgres, h2, rocksdb and quite a few others still seemed to take the combination of a lot of data manipulations with only a bunch of RAM and the rest on HDD just fine.

But I don't suppose switching away from SQLite would be easy, right?

> so all I can really work on is a system that lets people head towards what they would prefer

Yea. Maybe actually just do what seems easy and right to you on the PTR -like dropping the underscores for spaces and doing double underscores or whatever, and then write the parsers to respect that so most submissions already are good.

If it turns out people really want different tag formats, maybe there should be a special "sibling" / "parent" - like category (but separate from the manual one) to maybe take scripting generated mappings. Something that can not mix with the manually entered data and thus be dropped and re-generated with new scripts at will.

> My fingers-crossed hope is that neural nets will make this all a lot easier to manage in the coming years.

Might happen, sure. Maybe your PTR will help train these.

I occasionally get these popping up back to back whenever I search for a tag. Other than that I don't notice any problems.

AttributeError
'AutoCompleteDropdownTagsRead' object has no attribute '_dropdown_hidden'
  File "include\ClientGUI.py", line 3442, in TIMEREventPageUpdate
    page.TIMERPageUpdate()
  File "include\ClientGUIPages.py", line 771, in TIMERPageUpdate
    self._management_panel.TIMERPageUpdate()
  File "include\ClientGUIManagement.py", line 3345, in TIMERPageUpdate
    self._UpdateCancelButton()
  File "include\ClientGUIManagement.py", line 3202, in _UpdateCancelButton
    self._searchbox.ForceSizeCalcNow()
  File "include\ClientGUIACDropdown.py", line 583, in ForceSizeCalcNow
    self.DropdownHideShow()
  File "include\ClientGUIACDropdown.py", line 383, in DropdownHideShow
    self._ShowDropdown()
  File "include\ClientGUIACDropdown.py", line 322, in _ShowDropdown
    if self._dropdown_hidden:


AttributeError
'AutoCompleteDropdownTagsRead' object has no attribute '_dropdown_hidden'
  File "include\ClientGUI.py", line 3442, in TIMEREventPageUpdate
    page.TIMERPageUpdate()
  File "include\ClientGUIPages.py", line 771, in TIMERPageUpdate
    self._management_panel.TIMERPageUpdate()
  File "include\ClientGUIManagement.py", line 3345, in TIMERPageUpdate
    self._UpdateCancelButton()
  File "include\ClientGUIManagement.py", line 3202, in _UpdateCancelButton
    self._searchbox.ForceSizeCalcNow()
  File "include\ClientGUIACDropdown.py", line 583, in ForceSizeCalcNow
    self.DropdownHideShow()
  File "include\ClientGUIACDropdown.py", line 383, in DropdownHideShow
    self._ShowDropdown()
  File "include\ClientGUIACDropdown.py", line 322, in _ShowDropdown
    if self._dropdown_hidden:

Dunno if it will be fixed in the new release that will come up in a few hours, but after Corrupt EXIF data it stops downloading altogether.

Causing the the "Waiting for subs to finish".

2018/02/28 19:03:50: /media/hydrusnetwork/PIL/TiffImagePlugin.py:742: UserWarning: Corrupt EXIF data. Expecting to read 2 bytes but only got 0.

>>8162

Regarding the pausing due to an error. I stated it in >>8161, if the client gets "UserWarning: Corrupt EXIF data." while doing subs, it will freeze the sub, prevent the client from closing, and after a forcefull shutdown set the "paus subscription syncronisation" to true.

>>8164

👌

>>8161

>>8167

Thank you–sorry, I missed this the first time around. I will check this out this week. Can you give me an example sub search or a specific page that will give a bad file like this?