/hydrus/ - Hydrus Network

What's our progress looking like? Curious how close to done we are..

>>651

Hard to properly estimate as they are currently fixing up a lot of early missing blocks. After these missing submissions, of which there is around a million (i'm not sure), we'll continue from submission 12500000.

Seems like I was a little optimistic with my estimate previously.

My hosting is going to be doing an software upgrade between the 28th and 29th. They have said this might cause some network issues, but it should hopefully not be that large a problem, but I'm going to be observing the network activity to the best of my abilities in that time period.

I'm also currently looking into getting the AWS trial to add some speed to the network, but this is low priority right now.

>>653

Sounds good. Currently I am thinking that since we're doing the whole mess at once, I'm going to run one massive "non-ns tag unification process" in the sanitizer once we've got the whole dataset in hand. Instead of doing it all manually I'm going to sort out about the top 10% of erroneous tags and correct for them; the rest will have so few counts as to be functionally irrelevant.

Shit, they made an change in their default layout out of fucking nowhere. I am not in a position to fix this right now, so I shut down the server.

I also need to investigate a better way to save the state of the server. It has constantly been avoiding saving progress when shutting down.

I'll look into both tomorrow.

>>677

Thankfully, they have reverted the layout change after mass outrage, so we can probably start up again.

>>681

Sorry, 'real life' have been really hectic this last week, I've barely got time to work on stuff.

Turns out that when they changed the sidebar they also made some arbitrary change in the html for the title and creator display. This was the reason I didn't just restart the server when I noticed that they changed it back.

I fixed it by simply changing the code to extract it from the page title. Simpler and hopefully a little more future proof.

Has this been sanitized yet?

It would be awesome to have at least a FA-based artist:-tag database

Oh! I forgot about this project. I'm guessing no news for a year is a bad sign? A database like this would be so damn useful.

>>2621

cool, thanks for the reply.

I think I'll wait for the update

what you are doing with this program is really cool and I hope one day I can do something of similar scale.

learning java right now

>>2567

This site hosts backups of FA.

https://vj5pbopejlhcbz4n.onion.link

>>2621

Can I make a suggestion?

It's something that's been bugging me for a while - I really REALLY like the idea of a global tag base, but I despite how there are so many bullshit tags.

Would it be possible to affect it in such a way that the public repo only contains tags that get a lot of hits to it? For example, show tags. Obviously, useful.

I've put in a lot of requests to this in an effort to help clean it up, but it's amazing to me how many tags get added to the repo that are nothing but garbage. (Tags containing just a character happen a lot.)

I think the purpose of a general tag repo should be to apply general, but useful tags. It certainly kind of sounds like a pain in the ass, but I honestly think it might go a lot farther in just ensuring that it keeps the repo clean.

I'm just curious as to how many single tags out there exist. Certainly plenty of tags with different names that aren't in the same group. Just something I've been tossing around for a while.

PS - I let the whole thing sync in it's entirely, so whenever I go to add a tag, it takes a moment to search through everything. It's a slight pause, I've gotten used to it. Is this normal, or is it a bad symptom of letting the whole database sync?

>>2538

>>2541

I was the guy doing the sanitizer for this, and no, I haven't cleaned the final DB because life got in the way, among other reasons (there are some data integrity issues resulting from the process we used). Having become a much better coder in the intervening time, I recently scratchbuilt an e621 tag ripper and posted the cleaned result DB in the big database thread; many of the patterns I used for that can be applied to FA, and I just need to find the time to sit down and write the code for it, at which point I'll begin re-ripping the entire site's metadata.

>>2567

Unfortunately no - there is no method by which we could include a thumbnail in an HTA, and even if we could there's no way for us to get thumbnails from deleted subs, because FA deletes them when the sub is deleted (as far as I know). >>2631

is your solution here; it's slow and painful, but you can in fact do mass-downloading from it given the appropriate TOR-friendly tools.

I have also run into the problem of "artists delete stuff I probably would have liked"; my solution to this was to write a Python script that runs daily and automatically downloads all new submissions directly to my computer, then nukes them. This way an artist has to delete something within 24 hours, or I will have already grabbed it. Of course, this further aggravates my "I have too many images to sort through" problem, since I'm watching close to 1600 artists which leads to ~300 new files a day…

Code for that will eventually be posted somewhere, probably along with the tag ripper code once it's written.

>>2679

Okay, replying to this got my coder juices flowing, so I hammered out a parser for FA pages and will start scraping data; once I have enough data I'll re-attack the sanitization problem.

Request for public comment:

Below, as an example, is all the metadata I can think to extract from http://www.furaffinity.net/view/19948852/ :

{'artist': u'feralise',
 'category': u'artwork (digital)',
 'faId': 19948852,
 'gender': u'male',
 'keywords': [u'feralise',
              u'adult',
              u'sketchem',
              u'pyro29',
              u'pyro',
              u'vi',
              u'navalia',
              u'arctic',
              u'white',
              u'wolf',
              u'smirk',
              u'glance',
              u'erection',
              u'boxer',
              u'briefs',
              u'chill',
              u'temptation',
              u'muscle',
              u'pose',
              u'sexy',
              u'male',
              u'canine'],
 'postTimestamp': 1463018547,
 'rating': u'adult',
 'sha1Hash': '6ad603c975d20600bb565f9619ac5c51647feffd',
 'species': u'canid - wolf',
 'taggedUsers': [u'pyro29'],
 'theme': u'general furry art',
 'title': u'chill'}

Keeping in mind that all of this is completely raw, since processing and sanitization happens at a later stage (also, ignore the "u'" bits, that's just a Python marker for "unicode string"), can anyone think of any additional data that is available from FA that I am not gathering?

did my posts get deleted or is 8chan being garbage

>>2684

>>2709

Oh yeah just cache garbage.

>>2710

Godsfuckingdamnit there's a captcha on the FA login page now, for *every* login.

It will take me a not insignificant amount of time to figure a way around this.

>>2784

Ah hells. I was just about to mention that personally I would need just the artist/species for tags while everything else would be a bonus, but now that seems pointless. That whole leakage deal has been a pain.

I've actually figured it out, I think - mechanize can use the session key from my browser login to avoid having to log in itself. Project is back on once I have time.

It's been a couple months since there was an update on this, is anyone still working on getting a FA database going?

If this is dead, perhaps a new project could be started for https://beta.furrynetwork.com/ ? Seems to be the "new" Furry art site and will probably be hosting more and more of FA's content as artists move over to it.