[–]▶ No.980625>>980643 >>980651 >>980706 >>980726 >>981025 >>981064 >>981066 >>981445 >>981744 >>981856 >>981921 >>981931 >>982797 >>983015 >>985043 >>985185 >>985950 >>988653 >>989176 >>1008797 >>1022051 >>1022900 [Watch Thread][Show All Posts]
I'm one of those people who saves images off of *booru-type boards, and while I tried to keep it organized I inevitably found myself with folder like "good1", "good17", "abs", etc.
Well, I noticed my sloppy habits were using lots of HDD space, so I wrote a little bash script which would put all the images in one central folder, md5 them, then link them in my categories. Saved some HDD space.
But then I f'd up, and accidentally scrambled my folders. With everything in one big fat disorganized folder with md5 names I really didn't want to put effort into sorting them... ... So I wrote another script which would scrape various booru sites and get the tags for the images I had, dumping them into a text file. I didn't know what to do with it, but I knew it was a good step.
A week later I went back and made another script that would create and populate folder based on searches I threw at it. I could just run something like "prondir big_breasts flat_chest -o bigandsmall", and it worked nicely!
But this was ugly. I like nice things. So this morning I started the bare bones of a proper SQL database, tossed a custom MVC CMS I wrote into it, jammed my tag database in there, and put together a mini-site for my glorious porn browsing.
Then I made it nicer... and I started adding features. I crossed the event horizon of "this is for me" into "hey, I think this could be a thing?" It's only day 1 into "formal" development, but I re-wrote my Bash stuff into PHP, and started on features that I would want. E.g; when you view an image it will also recommend images you might also be interested in.
So, if I keep going on this thing - what would anyone want to see in a new booru board? Since it's only got bare bones, there's not any tech debt, planning now around what everyone wants seems like an idea.
▶ No.980631>>981005
sudo rm -rf / should fix the problem
▶ No.980637>>980638
Post screenshots of how it looks so far
▶ No.980638
>>980637
shit never mind I'm an idiot, didn't see images until after I posted
plus no PW so slim jim can't track me
mods delet my post plz
▶ No.980643>>980664
▶ No.980651>>980660
>>980625 (OP)
>what would anyone want to see in a new booru board?
support for in-line images.
I want long form posts like you'd find on Usenet or 90s forums, with inline image bbcode-esque support, but with the ease and anonymity of posting on a chan.
Make it happen.
▶ No.980655>>980665 >>982086 >>1004769 >>1010714
>furry
>loli
>traps
>PHP
I do appreciate your post, /tech/ needs more people who actually do stuff, but you should seriously consider suicide.
▶ No.980660
>>980651
So what I'm thinking with that (just to prevent abuse) is to allow only on-site post images to be inserted. E.g. inserting a 1px transparent remote image that tracks you, or swapping out an image later, or simple breakage. I could add remote images anyway (it's not hard), but in whatever default configuration it might come with I'd disable them by default.
But yeah, Ive done BBCode stuff before, no reason not to have all the basics.
On an aside, users will have "albums". I was always going to have them so you can add images to albums and run on-site slideshows; but maybe I could also give users a "stickers" album people could use to collect reaction images and the like, with a "quick insert" button when posting offering those images. Sort of like how you can collect them on Telegram.
▶ No.980664>>980668 >>980669 >>980743
>>980643
>>>>>>>>>>>>>>>>>>>python
tmsu is better
▶ No.980665>>980874
>>980655
So what if I want to fuck PHP and code in furry!?!
▶ No.980668>>980743 >>981742
>>980664
Tbh, this is actually one time where the >python meme is right
Hydrus is a slow bloated piece of shit.
▶ No.980669>>980672 >>981888
▶ No.980672>>981742
>>980669
'cept it's not loading a full fucking gui in 1000.
It's a simple script that integrates with your regular browsing.
It works fine here.
▶ No.980706>>980818
>>980625 (OP)
fucking furies haha
▶ No.980726
>>980625 (OP)
Yeah, I've been thinking of doing this for years. Kinda started on it. Made the symlink tagger, and made a search function. My idea was instead of md5ing the files in the pool, I'd md5 just the search results, so you can post them somewhere else without leaking trackable names, while it still has whatever name you want on your system. It's something I never got around to, like most of my programming projects. github.com/SpaceBudokan/SackyTag
▶ No.980743>>981742
>>980664
>>980668
I haven't been able to figure out why anybody would use python for web dev, but then again I have no idea why people use php either.
▶ No.980831>>980976
>md5
Why not use an algo specifically designed to match images based on visual similarity? There are several.
▶ No.980832
▶ No.980874>>980927
>>980665
>furry
Sorry guy, but after hearing about that dead dog fucker idk if i can tolerate furries anymore.
▶ No.980927>>980939 >>981165 >>982749 >>983919 >>986028
>>980818
>>980874
>Because one person did something fucked up, that means everyone in said subgroup is equally as guilty.
Sweet logic bro
▶ No.980939>>980970 >>981022
>>980927
Furry detected...
Even if not, while it is an overgeneralization in most cases, it isn't in this case given all the other ridiculous shit from the furry community, including (but not limited to) plushie sex, vore, beastiality, diaperfurs, and so on...
It is a very safe bet to say that whole fandom is flaming trash.
▶ No.980965>>980967 >>980990 >>989990
Maybe off topic? But on the subject of...
>put all the images in one central folder, md5 them, then link them in my categories
I've played this game. Dir reads take a mega long time with tons and tons of files in them. I've found in my testing when I did something like.
#pseudo code
sum = generate_sum(filename)
dirname = s[0]s[1]/s[2]s[3]/
mkdir dirname
mv filename dirname
And I don't know if you're interest in this but noting that visually identical (down to the pixel) images can have different md5sums due to so many different variables that simple cryptographic hashing isn't sufficient for something like this. You're going to need to do perceptual hashing if you care about that. At the very least you can turn the image greyscale and scale down to a reasonable size you can easily store and compute a hamming distance against other hashes. That will catch less false positives than you may think.
Here's a shitty little testing thing with hardly any functionality I did a couple of years ago that should illustrate what I mean. https://0x0.st/sYpT.tgz
It's what I used to generate 5.png from 4.jpg. 5.png just visually represents a 64byte hash that can be stored in a file name perhaps.
These links may hook you up.
http://phash.org/ <- this is what I use in my personal project.
http://bertolami.com/index.php?engine=blog&content=posts&detail=perceptual-hashing
http://blockhash.io/
▶ No.980967
>>980965
>64byte hash
Meant to say 256
▶ No.980970>>980974
>>980939
>Furry detected
Try again, smartass.
The furry fandom is way too vast to be generalizing like this.
▶ No.980974
>>980970
In a way, they kind of asked for it- with their constant attempts to absorb other fandoms and the like, as well as the rejection of some anti-bestiality people in the earlier days. Not going to judge them based on zoophiles hiding int heir ranks though.
▶ No.980976
>>980831
It's just for naming the images, if I wanted to scrape tags from other sites I'd need to md5 them at some point. It's also a decent way of avoiding name collisions with uploaded material.
▶ No.980990
>>980965
I did something like this back in the day. What I did is downscale the image to ~50 pixels of data while logging the ratio. After that I converted the image to a limited indexed palette I created. Then it was simple to store the images as strings and compare them. The one requirement was that the image have roughly the same aspect ratio, within a certain tolerance, for the algorithm to be computationally light on larger imagesets.
▶ No.981003>>981014 >>981254
People bitching about your porn taste OP, but good for you for doing something productive. Was wondering how fast that textfile was over the SQL database? I know it'll be slower but I don't mind waiting a second or two for the sake of not having to set up a local server on my toaster of a machine. I've thought of doing something similar for my own image folders, but I pull images from so many places I'd have to manually tag like 20GB worth of shit and gave up.
▶ No.981005
>>980631
suicide might also be a fix.>>980631
▶ No.981014>>981015 >>981020 >>981742
>>981003
It was actually pretty respectable. For a 10.1GB folder (5534 files, not including video) running on my HDD, all operations seemed pretty much instant. That being said, I also have a strong i7 CPU.
I even had the "you may also like" feature working through the text file, and even comparing literally every tag in every image it was 'instant'. Had to be < half a second. But I'm also sure the database performance would slow down exponentially with larger datasets... Not sure. You must be hitting ~10,000 images.
The downside is that there was a limit to what I could do with only the text file. E.g. tag aliases, view counting, etc. That being said, probably no reason it couldn't have a SQLite back-end. I'm focusing on getting things running and fleshing out the DB, but the core CMS underneath could eventually make that possible. It will have a comfy installer, I like comfy installers, so no matter what setup should be painless.
▶ No.981015
>>981014
I tested the text-driven model again. It generated the heaviest page in 0.267 seconds cold, and 0.060 seconds hot. So performance is decent.
▶ No.981020>>981021 >>981025 >>981411 >>981742 >>985185
>>981014
So here's a screenshot of a "tag-heavy" image in the old text version. Specifically in the "you may also like" it's comparing all the tags in every image, doing a formula, and those numbers are the "weight" of the matches. In the bottom left corner it shows the generation time, to to compare every tag in that image with every other tag in every other image, it took 0.041 seconds for that one.
▶ No.981021
>>981020
Shoot, sorry for not spoilering that; mods - could you help me out? Sorry.
▶ No.981022
▶ No.981025>>981031 >>981066 >>981742
>>980625 (OP)
>>981020
I use hydrus but his viewer is honestly shit. Not to mention all the times that it fails to grab booru tags after scraping.
The similar search at the bottom looks like it would be a good feature for any booru like board to have so why not? Are the numbers at the bottom based on a best match and decline from there? How does the "You may also like" algorithm work?
▶ No.981031>>981039 >>981929
>>981025
I'm not sure why it isn't a super-common feature either. This is how my own first stab roughly works:
1. Find all images with at least one tag in common with the current image to create the sample set.
2. Inversely weight all tags in the image by their commonality among the sample set. 1/sqrt(totalOccurencesInResults). This makes it so 'rare' tags (e.g. a character name or distinctive feature) will give higher weights.
3. Sum up the weighted tag scores for each sample.
4. Divide the score by square root of the total number of tags in that image. This is so images with large numbers of tags don't steal the limelight from more relevant images.
5. arsort & slice.
There's a couple little fiddlybits I left out, but that sums it up. By the looks of it, "great matches" seems to hover around a score of ~3.1+, O.K. images are ~2.8, and it drops off from there. I (think) the system will get more accurate with a larger database though.
This is still mostly done in PHP, but I'm wanting to make it more SQL-driven.
▶ No.981039>>981051
>>981031
sounds good to me,rare tag having more weight is smart. Hydrus or other boorus use categorization for characters and series which is a simple lock but but how would you differentiate a distinctive feature from the rest of the noise.
▶ No.981051>>981742
>>981039
I've been thinking about that. In the DB all tags have taxonomies in addition to types already.
Instead of complicating the general "you might like" formula, other things could be done. Probably down the road. I'm getting somewhat happy with the recommendations it's giving with the formula tweaks. It sounds obvious, but the more you complicate a formula the harder it is to get predictable results - and it's near impossible to adjust a complex formula without reverberating repercussions.
Instead I'm interested in integrating taxonomies and tag parentage to improve the exploration level of the system. For example, if you search for "pink_skin" I'm aiming to have it suggest stuff like popular characters with that trait, like "majin android 21" and "emelie" if it finds strong correlations. Maybe there's an author who draws em' thick, and it might suggest that author when it picks up on what you're looking for.
▶ No.981064
>>980625 (OP)
Princess Zelda by Didi-Esmeralda.
▶ No.981066>>981071 >>981073 >>981742
Download when?
>>980625 (OP)
>>981025
Is there even an option in Hydrus to scrape tags for multiple images at once? I somehow still can't find it. I'd use OP's thing for that alone and even if it doesn't Hydrus doesn't seem to support multiple databases and I would like one for porn and one for memes. I like using imgbrd-grabber to browse and individually download the best pics over downloading entire tags and it doesn't seem to have an option for saving tags besides in the filename. (which isn't long enough)
▶ No.981071>>981074 >>981742
>>981066
>Is there even an option in Hydrus to scrape tags for multiple images at once?
Isn't this one of the main features of hydrus? It's nuts how much file bloat you can get because you threw in a general tag to search for. Thats why there's a file limit set up so you dont download 2gb in one sitting.
>Hydrus doesn't seem to support multiple databases and I would like one for porn and one for memes.
Default hydrus is set up for one database but more than one user, including the dev of hydrus has done more than one db.
▶ No.981074>>981107
>>981073
>Copyright 2011-2018 Paul Ruane
You've been doing it for that long? Or is this a fork or something?
>>981071
Maybe I'm retarded after all. What do you mean by more than one user though? As in actually making a new system user and running it with su/sudo?
▶ No.981107
>>981074
i mean more than one anon has done it. not that you need a different user system since there's no accounts. the hydrus help has written up on making more than one db. i never did it though, just decided to split the db load between drives.
▶ No.981112>>981124 >>981127 >>981464
>>981073
>TMSU
This isnt OPs project and you're a faggot. tmsu is a meme program but with little practical use unless you want to be anal about every file. it doesnt come anywhere near hydrus's PTR functionality. Who the fuck wants to manually tag tens of thousands of files?
▶ No.981124>>981129
>>981112
>manually tag tens of thousands of files
>i'm a brainlet and can't into shell scripts
TMSU actually provides more functionality than hydrus, sense it uses command line options instead of a bloated gui.
▶ No.981127
>>981112
Also, who the fuck keeps shilling it? I've seen it in like 3 threads today.
▶ No.981129>>981131 >>981464
>>981124
>I'm a retard who doesnt understand what a PTR is
you dont need to tag thats the point of the feature you stupid ass shill.
▶ No.981131>>981132 >>981141 >>981742
>>981129
>faggot doesn't understand *nix
Your project is shit.
It's not integratable into normal filemanagers at all, it's made in python, it's bloated and slow.
TMSU manages to do all of what hydrus does and more without any of the draw backs.
If someone wanted to use tag sorting in hydrus they're going to need to learn the hiddious ui that looks like it was made for fucking windows 95.
TMSU uses fusermount so all you need to do is use the same filemanager and media players that you've always had.
The entire reason why hydrus is shit is because it's not simple, or does the single task it's supposed to in a way that's easy to understand.
On top of that, once you actually learn how it works, you still can't do much around it.
Like i said previously, you can use TMSU with bash or any other scripting language you want to make everything nice and automatic.
You can't do that in hydrus.
▶ No.981132>>981133
>>981131
>Your project is shit.
Stopped reading there.
▶ No.981133>>981742
>>981132
>hyrdus is an 8chan project
>dev used to post on /tech/
I'd say the chances of you being the maintainer of it is quite high tbh.
▶ No.981141
>>981131
Takers shouldn't talk like that to Makers.
▶ No.981146>>981514 >>981548 >>981742
Having image sibling/alternates support would be a big help. That's a huge feature missing from hydrus right now. I've tagged thousands of images as alternates of each other using its phash deduplicator but it's not able to use any of the pairings. It's also advisable to integrate a REST api so other frontends can be plugged in if wanted/needed. Hydrus is a cumbersome monolith with an overly complicated GUI.
As a side note I've been planning on making my own booru browser too in a month when I get some free time. I want to experiment with ArangoDB to see the advantages of graph databases. I'm interested in hearing your progress with the recommendation algorithm. Adding namespace support might help. As a test set I have the 1.8TB Danbooru 2017 archive along with all the images I've downloaded on my own. Let's see what we can do with 2.9 million tagged files.
▶ No.981165
>>980927
You know its true.
▶ No.981193>>981256 >>981742 >>981799
Improvements!
There's a spangly homepage now... and a logo!
"You may also like" is much better. I added stats & info to the debug output, and with that I re-wrote the logic since it was easier to see what it was doing. I'm shocked at how well it's actually working now. When I re-scrape the tags I know a few more improvements to do. Scores are now normalized and offered as a % match, which also means the system now discards based on a score threshold so it doesn't serve vague trash. Posts below a certain number of tags (5) are not included in results, and don't show recommendations.
I've started a proper query parser which can handle a few things, like OR pipes (|) so you can search something like "breasts|ass".
▶ No.981254>>981256 >>981742
>>981003
>Autistically categorizing your jewish mind poison so you can waste your seed to jewish mind poison
>Productive
▶ No.981256
>>981193
Good shit OP, keep it up.
>>981254
>oh fuck it's not a C vs Rust, Lisp autists or blackpill faggotry
>SAGE SAGE SAGE
▶ No.981411>>981480 >>981664 >>981742
>>981020
Why are some tags green? Are those your favorite tags using the + button? If so it could take those into consideration when recommending images.
▶ No.981426
Very nice OP, I've been wanting to tag all my images for a while and considered using hydrus at one point, but I'd basically have to manually tag 18,000 images if I wanted to go beyong just using existing folder names. Some sort of basic booru lookup to get tags for an existing image would be excellent.
▶ No.981445>>981486 >>981514
>>980625 (OP)
How much space would it require to archive, say, Danbooru? I archive tons of shit and I'm afraid it'll suddenly dissapear someday.
▶ No.981464
>>981129
>>981112
It's trivial to use TMSU with PTR functionality if you know what you're doing. It's trivial to either use the hydrus libs directly or program something yourself to iterate through files and auto-tag recognized ones.
▶ No.981480
>>981411
That was just me fucking with tag attributes, but I like your thinking! I made a glyph font and made sure to include the <3 glyph, maybe I could put it to use.
▶ No.981486
>>981445
I can't really give any hard numbers at this point. I know the data storage is negligible, thumbnails are a big heavier than current booru boards (~15-20kb), and I'm not using downscaled images yet... those will affect the storage quite a bit.
▶ No.981514
▶ No.981548
>>981146
+1 to this.
parent/child images for edits or alternates is definitely needed. Also the ability to differentiate between an official artist edits or fan edits would be nice.
▶ No.981592>>981594 >>981596 >>981604 >>981742 >>981809 >>981839 >>981854 >>982042 >>983126
I'm a fucking dumbshit. I just accidentally overwrote all my jpgs with their thumbnails because I forgot a slash in my old script.
▶ No.981596>>981614
>>981592
congratulations anon, you are the proud new owner of three terabytes of furry porn for ants
▶ No.981604>>981607 >>981614
>>981592
At least you know where to find the originals, right?
▶ No.981607>>981614
>>981604
He said he overwrote the pictures.
▶ No.981614>>981648
>>981596
>>981594
>>981604
>>981607
Well, it seems only the images in the database were thumbnailed (... which makes sense), so I took it as a challenge to add an image downloader to the system. I have a temporary one running and recovering things now - though I can't tell if I'll be able to recover images only found on rule34.xxx. They have stupid urls.
But so far it seems like I'm recovering about an image a second, so that's fine for me.
▶ No.981648
>>981614
Recovered all but 108 images! XD
▶ No.981664
>>981411
I thought the +/- buttons were for the tag search only. Like how e621 does it, with adding the tag to the current search filter.
Except e621's is halfassed and it's only as good if you are too lazy to type it in yourself. But it isn't keeping track of the current tags like a checklist filter on some store sites. So if you add +tag then -tag they will just cancel eachother out and you'll be left with nothing.
▶ No.981742>>981758 >>981811 >>981831
>>980668
>>980743
First, try Hydron (written in Go) https://github.com/bakape/hydron
Then, understand that Hydrus is rapidly prototyped for the last 5 years to support thousands of websites.
>>980672
Not enough features.
>>981014
SQLite is always the answer for speed
>>981020
Looks good
>>981025
> "You may also like"
Hydrus Dev is implementing it in the near future
>>981051
What do you mean by "taxonomies"?
>>981066
>>981071
Learn 2 get scripts https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts
There are also default tag scraping for the major *boorus
>>981131
> looks like it was made for fucking windows 95.
Then rewrite the code to use Qt instead of WxPython whiny bitch, or make a CLI if you want things to be fast
>>981133
Nah, the fanbase is large.
>>981146
> Having image sibling/alternates support would be a big help
Already implemented
> I've been planning on making my own booru browser too in a month when I get some free time. I want to experiment with ArangoDB to see the advantages of graph databases. I'm interested in hearing your progress with the recommendation algorithm
Go on, make it happen.
>>981193
Put it on GitHub/GitGud
>>981411
Tag characteristics
>>981592
Hydrus WINS!
>>981254
>>>/oven/ >>>/trannypol/
▶ No.981744>>981745
>>980625 (OP)
Why are you not in an institution?
▶ No.981745
▶ No.981758>>981773 >>981774 >>981799
>>981742
>What do you mean by "taxonomies"?
In addition to types (character, author, copyright, meta) I'm adding taxonomies, Taxo will describe what the tag is describing. For example, "black_and_white", "3d", and "photo" would go under under the "style" taxonomy. "sleeping", "screaming", "running" would have a verb taxo. "2girls", "solo", "female/male" would fall under a subjects taxo, etc. It will also differentiate tags of the same name a little more nicely; is the image a photo, or is there a photo in the image?
Essentially, tag type will be required for a functioning tag, taxo will not be, but taxo will in general help a lot of things. When all the types of taxo are set in stone, images will require tags with certain taxonomies before they can be fully approved. Every image needs a noun, every image needs a subject, a verb, etc.
>Put it on GitHub/GitGud
I have to go through and strip my name out of the source headers first; I borrowed a lot of code from another project, and this isn't something I want to associate myself with. Future employers might get squeamish, so it will be developed under my "18+" alias "Dizmal". :/
▶ No.981774
>>981758
I also didn't get what you meant but now I get it. I find illustrating ideas helps a lot. Basically it's tag relationships. You can have top level categories and have tags branch from them.
Category1
/ | \
Tag1 Tag2 Namespace:Tag3 Tag4
The image siblings would be at one level but could be combined with parent/children hierarchy.
Sibling1 --- Sigbling2 ---Sibling3
Parent
|
Child1&Sibling1 --- Sibling2
▶ No.981784
>>981773
What are you talking about? None of the source code I have has ever been public. I also never said where specifically I'd be publishing the code.
▶ No.981799
>>981758
Then Hydrus already has taxonomy, but you should try make your own.
>>981773
Not talking about TMSU, fag. I am talking to Dizmal's Borehole (make a better name please). >>981193
▶ No.981809
▶ No.981811>>982019 >>985436
>>981742
>Hydron
alright, i'm listening.
Will this have cli arguments?
▶ No.981831>>982019
>>981742
>how do i use scripts on more than a single file at a time
>lol just get scripts
???
▶ No.981839>>983126
>>981592
Shit like this is why I have everything important on zfs, with automatic snapshots using pyznap or sanoid. Instant and painless rollback.
▶ No.981854
>>981592
I tried to write a script to detect and link duplicate images. Somehow the script didn't save the hash of the new images properly, so any duplicate file was replaced with the first duplicate file the script found. (After recovering the files I could, I ended up deleting that image entirely because I was so sick of seeing it.)
▶ No.981856>>981881 >>981894
>>980625 (OP) To this booru please op. I wanna look at that sexy dog pussy porn in the search results there.
▶ No.981881
▶ No.981888
▶ No.981894>>981909 >>981941 >>982325
>>981856
It will come... In the meantime...
IT NOW HAS SEXY PORNHUB STYLE VIDEO THUMBNAILS ON HOVER!
Huehuehaha, sample attached. Though they have the "gifv" extension on the site.
▶ No.981903
Hype hype hype!
BTFO FIoCoder
▶ No.981909>>981912
>>981894
Oh? Do tell us now.
▶ No.981912
>>981909
It uses FFMPEG and FFPROBE to do the heavy lifting. To make the thumbnails it takes & scales 5 2-second slices from the video, then saves the concatenated result as a gifv with webm encoding.
For the static thumbnail it takes a still from the half-way mark. I hate it when videos fade in from black at the beginning, making the thumbnail a useless black rectangle.
▶ No.981921
>>980625 (OP)
Although i like the idea of hydrus it fucking chugs on my thinkpad, if your implementation supports mass-scraping media from booru sites with tags I'll give it a shot.
▶ No.981929
>>981031
>1/sqrt(totalOccurencesInResults)
Are you using Q_invsqrt?
▶ No.981931
>>980625 (OP)
I'll forgive your terminally shit taste in porn on account of the fact that this is more programming than /tech/ has done in years.
▶ No.981941>>981952
▶ No.981952>>981982
>>981941
https://e621.net/post/show/1623850
There are other angles in the description.
▶ No.981982
>>981952
Why does this shit always look but better in the thumbnails?
▶ No.982019>>982029
>>981811
Just ask the dev, he will probably cave in to your demands, it is the same goy who made Meguca.
>>981831
It is on autopilot, so it scrapes all tags with the image by default once you install the all-in-one script (or not if it already have the scripts installed, check network->downloader and network->downloader definitions). If you are talking about the case of "import image from other places but forgot the tags" then that gets a bit tricky.
▶ No.982029>>982051
>>982019
network→downloader doesn't exist and network→definitions doesn't do anything. Is something fucked up?
>If you are talking about the case of "import image from other places but forgot the tags" then that gets a bit tricky.
Yeah, I have a number of images that I know came from certain boorus but don't have tags from them. So basically pic related but for a large group of files
▶ No.982042>>982051
>>981592
and this is why you always keep backups
▶ No.982051
>>982029
Top navbar, file-pages-database-network-services-help, select network->downloaders->import downloaders for installing scripts, network->downloader definitions ->manage [anything] for checking out what is already included
>>982042
And this is why he should use Hydrus with ZFS
▶ No.982083>>982084 >>982091
(25)
And OP still hasn't snipped a line of his shit code.
▶ No.982084>>982085
>>982083
Probably still better than the hydrus shitshow
▶ No.982085>>982320
▶ No.982086
>>980655 (check'd)
Dubs of truth.
▶ No.982091>>982324
>>982083
$post = \Model\Post::getPostById($id);
▶ No.982125>>982133 >>982149
Oh hey a thread where some anon is actually doing a thing
>it's software to sort his enormous collection of furry, pony, loli and trap porn.
▶ No.982133>>982320
>>982125
Again, >>>/hydrus/
You ain't seen shit
▶ No.982149
>>982125
He has all the biggest meme fetishes. What's next after that? /cuteboys/?
▶ No.982320>>982329 >>982357
>>982085
>>982133
Shilling your pajeet code like this isn't going to make anyone want to use it.
▶ No.982324
>>982091
Ok, now post something that we can install and use.
▶ No.982325>>982344
>>981894
Can you add the option to import files as symlinks instead of making second copies of them? It's a feature that nothing else has that I really want.
▶ No.982329
>>982320
open Bob and show vegana sexi girl
▶ No.982344>>982360
>>982325
That might be nearly impossible to avoid depending on your system. Since this is a web-based imageboard you need to blow a giant hole into security settings to display images.
Now, if you're stupid and aren't encrypting your porn, no problem! Symlinking will work fine.
Either way though, I think I need to address the parallel conversations going on; I'm developing an online imageboard, not a desktop application. I'm sure a lot of the features I'm working on (easy installer, built-in slideshow, personal albums, etc) will make desktop usage attractive, but that all assumes you already have a local webserver.
More/less I think the imageboards online today kinda suck experience-wise. The tech under the hood is probably pretty good in many cases, but actual day-to-day usage feels like a product of the early 2000s.
...That being said...
I also develop with Qt and QML on database-driven applications. Looking at Hydrus, I'm 90% sure it's in the same boat - probably some good stuff under the hood, but that UI would have been considered bad in the 90s. I also get opinionated with desktop apps - write the core in a damn compiled language.
I poked my nose and downloaded ROFS to see what a FUSE filesystem might look like - it's nothing too scary. Maybe when Borehole has a more solid database structure and I need a break from HTML, I'll look at a desktop-oriented UI with a FUSE system as a companion project. Maybe I'll call it "Drillshaft" or something.
▶ No.982357
>>982320
Found the no-code PM
▶ No.982360>>982366
>>982344
>Now, if you're stupid and aren't encrypting your porn, no problem! Symlinking will work fine
What does encryption have to do with symlinks? As long as the encrypted filesystem is mounted it's functionally just like anything else. Are the files actually stored inside the database instead of the filesystem? If so why?
>I'm developing an online imageboard, not a desktop application
We can tell from the screenshots. That's not going to stop me from using it on my local network though. There's nothing wrong with using a web based frontend instead of a native desktop GUI. Hell, that's what we've all been using all these years.
▶ No.982366>>982372 >>983126
>>982360
>What does encryption have to do with symlinks? As long as the encrypted filesystem is mounted it's functionally just like anything else. Are the files actually stored inside the database instead of the filesystem? If so why?
I thought so, too! Maybe I misinterpreted something along the line while getting my stuff set up, but even with proper permissions Apache/PHP refused to read (and the browser refused to load) anything in my encfs folder. That's even after it was unlocked and should have behaved like a standard directory.
I ultimately had to add apache to my usergroup for it to read from the encrypted folder... So, big security hole.
▶ No.982372>>982387
>>982366
Why aren't you using dm-crypt? I'm assuming you're using the encrypt directory functionality of your desktop environment. What you're describing sounds like it might be an EncFS issue, not a general issue.
>I ultimately had to add apache to my usergroup for it to read from the encrypted folder... So, big security hole.
Why would that be a security hole? It needs read access one way or another. Are you sure you have your permissions set up correctly?
▶ No.982381>>982382
What's the difference between this and Hydrus?
▶ No.982382>>982388
>>982381
It's really simple, it's read the fucking thread.
▶ No.982387>>982392
>>982372
>Why aren't you using dm-crypt? I'm assuming you're using the encrypt directory functionality of your desktop environment. What you're describing sounds like it might be an EncFS issue, not a general issue.
Probably, but it's all running now, so it's good enough.
>Why would that be a security hole? It needs read access one way or another. Are you sure you have your permissions set up correctly?
I suppose it's not really, but I really dislike giving anything more access than needed. It's not like I'd be vulnerable anyway, but it's an old habit. You just don't know how people use/abuse their machines, so if my software requires any weakening at all, I work under the assumption I'm the last step in an emerging vulnerability. Probably paranoid, but it was literally my job to be paranoid at one point.
▶ No.982389
>>982388
Look at the screenshots in the OP. That's literally all it is, for all we know he made it in photoshop and hasn't programmed anything. Because he hasn't released anything "yet"
▶ No.982392>>982399
>>982387
>but I really dislike giving anything more access than needed
I'm almost positive what you're experiencing is a file permission misconfiguration. I don't know your experience level with file permissions but if you want to protect yourself I suggest you learn them. Your encrypted directory probably has 640 permissions which means only the owner can read and write, the group can read, other users can't read or write, and no users can execute files. Adding apache to your group is a secure way of granting it read access.
Now you can experiment with symlinks since they inherit the permissions of whatever they're pointing to.
▶ No.982399>>982426
>>982392
T'was the first thing I looked at, I started with 664 and went to fill on 666, didn't work. Probably an encfs idiosyncrasy, maybe even a security 'feature'.
I should also correct myself; I didn't add apache to my usergroup, I set apache to run under my user.
▶ No.982426>>982439
>>982399
>I didn't add apache to my usergroup
>I set apache to run under my user
Well there's your big security hole you were concerned about. Ironic.
▶ No.982430>>982434 >>982444 >>982526
I'm going through my code now and getting it ready to to put in a proper git repo. It's going to be a day or two, I think 90% of it all is going to be some really pedantic shit; making sure code formatting isn't a clusterfuck, finally fixing all the file headers, removing my name, etc. I don't want a tonne of my commits being "converted tabs to spaces".
I'm also using some old code, so I'm going to modernize it a bit. I had a few TODOs on "future features" I wanted to use to simplify the core code, and it's time I do that.
I figure I'll license it under the either the GPLv2 (without the "later"), or the GPLv3. If anyone has some input here, I'm all ears. I'm generally okay with the GPLv3 text, it's just the "or later" part I'm not a fan of, and that's required in the v3. I'd hate it if the GPLv4 went full RMS autism and impacted the project.
Also, what git service does everyone here recommend? I'm not going on Github; I'll jump on nearly anything as long as it's reliable and politics-free.
▶ No.982434>>982464
>>982430
https://gitgud.io or https://gitlab.com/ seem safe so far. Another one to avoid is bitbucket. Same poz, just less obvious if I remember properly.
▶ No.982439>>982444
>>982426
It's really not that of a big deal. Having read access over your home directory, which your usergroup usually does by default (in addition to having write permissions because default umasks suck), is a big security flaw on itself, but he would have that problem whether Apache runs as his user, or whether it runs as its own user (or FUCKING ROOT as is the case in many distros) with his user's group.
▶ No.982444>>982464
>>982430
Dual license it as GPLv2 and GPLv3. That way users have a choice of which license they want to pick.
>>982439
The proper way would to make the directory 640 and and change the group ownership to a newly created group. That way you add yourself and apache to that new dedicated group so it's still isolated from everything (assuming you had a sane umask of 640). Even then you can easily recursively change your home directory to that.
▶ No.982452>>982457 >>982464
Are you smart enough to implement a TAY type of ML program into your boards?
▶ No.982457
>>982452
As in TAY the twitter bot? What would the advantage in that be?
▶ No.982464>>982505
▶ No.982505>>982735
>>982464
Hilariously, if someone *had* to do something (like fix a bug) but didn't really *want* to, they would be in violation of the license if they went ahead.
▶ No.982526
>>982430
>it's just the "or later" part I'm not a fan of, and that's required in the v3.
Protip: it's not required
▶ No.982735>>982784
>>982505
What if they want to get it over with?
▶ No.982749
>>980927
Furries are subhumans and the literal scum of the earth.
▶ No.982784
>>982735
They want to minimize something they don't want to do. The want to do something is ultimately a FILO stack.
▶ No.982797>>982821 >>983040
>>980625 (OP)
>No more than 2 tags on danbooru
>No more than 6 tags on e621
Is there any reason why booru's have a limit on tag searching? Why the fuck is this a thing? I can't imagine this ever being a drain on the site performance. Please OP. If you make your booru public then don't follow in these nigger's footprints. Tags are there to be used as filters so why would anyone think it's a good idea to limit your filter?
▶ No.982821>>982884 >>983040
>>982797
I know, right!?!?! It drives me nads!
For standard tags I'm going to put it at something ridiculous (probably ~25 tags in the default config) and more/less that's just so some asshat doesn't try to run the entire text of War & Peace through as a query and hit a memory limit.
There will be a couple limitations though in regards to wildcard usage, mostly because some behaviors can impact performance in that department. I don't know what the plan is for that yet, though.
But yeah, search in these sites can be more capable. I think I already mentioned including OR pipes already. I'm flip-flopping between allowing brackets to group terms, because then with brackets and pipes you need to validate a query. Either way, among the 'features', I think not imposing low arbitrary limitations isn't that difficult.
▶ No.982884>>983040
>>982821
>25 tags
Thanks. that should be more than enough to search for exactly what I want. I've turned to scraping boorus and then using hydrus to get around this stupid limitation.
▶ No.983014>>983040
Question for the peanut gallery;
Would you rather have the source sooner (but without any way to actually use it) just to see some sauce... or... Would you rather wait another couple/few days (thank Canadian Thanksgiving) for something you could install and test to a limited degree?
This software is literally less than a week old, so don't expect any databases you might install to survive between any releases at all anyway.
▶ No.983015
>>980625 (OP)
why don't you just run your own imageboard at that point lmao if you're going to put that much effort into it.
▶ No.983026
>furry, pedo and ponyfag
Kill yourself.
▶ No.983040
>>982797
>>982821
>>982884
Hydrus + IPFS would make this problem go away. Or Hydrus or Hydron + Beaker/Dat, either way.
>>983014
"Talk is cheap, Show me some code" Linus Torvalds
"Release Early, Release Often, Listen to your customers" Eric S Raymond
"No problem is too big it can't be run away from" Linus Torvalds
"Prototype then polish, Get it working before you optimize it" Eric S Raymond
"Any program is only as good as it is useful"
"With enough eyes, All bugs are shallow" Eric S Raymond
"Those that can, do, Those that can't, complain"
▶ No.983126
>>982366
>encfs
This is a problem with how FUSE mounts work. Same problem happens with sshfs, not even root can browse it. There is the `-o allow_other` mount option to let other users browse them.
Encfs security is very questionable leaking metadata like mtime, file sizes and dir structure. It'll do the job preventing normies from finding your embarrassing furry smut, but won't stop the feds.
>>981592
>>981839 is right.
Use one of the many backup utils that use the send|receive feature zfs/btrfs (or at least an rsync frontend that uses hard linked snapshots if you are stuck on an old-world filesystem).
Another feature you get with zfs/btrfs is reflink copies. A copied file doesn't use any extra space until you write to it. Absolutely essential when you need to copy of a large amount of test images do you don't damage the originals.
▶ No.983134>>985186 >>1022069
https://gitgud.io/Dizmal/borehole
Just for you all I took a little time and got the text-driven post model working again. For anyone who wants to get a very basic picture of what it looks like, have fun.
▶ No.983444>>983456 >>983552
>all the images in one central folder,
Why do this, I am very reluctant to give up my folder structure into one big mess...
▶ No.983456
>>983444
this is your mind on UNIX
▶ No.983530>>983542
If you have balls try price-matching http://zenbooru.org/index.html
▶ No.983542>>983545 >>983546 >>983552
▶ No.983545>>984225
▶ No.983546>>983547 >>983552
>>983542
Okay, you use jQuery of all things... please refer to >>>/hydrus/9963
▶ No.983547>>983549 >>983552
>>983546
>>>983545
>>>983546
>I'm not the dev. I would sooner use windows than write javascancer.
I do however welcome a new booru that hopefully won't be as shit as gelbooru became.
▶ No.983549>>983552
>>983547
..I have absolutely no clue what the fuck happened with that formatting. I blame having to disable my referral blocker just to bloody post. Going to go re-evaluate my life now...
▶ No.983552>>983564 >>983584
>>983547
>>983546
>>983549
The main thing I care about is code readability; I don't use jQuery for compatibility, not for years now. I probably only even use a handful of utilities and conveniences.
I abhor jQuery plugins. I don't understand why you'd download a 100kb+ slideshow plugin, that won't perfectly fit your website, and will likely require half an extensions' worth of code to integrate... While doing 20 things you don't need it to do because it's not purpose-built for your situation. It's either laziness or deadlines that I ever used a plugin, and I was never happy with the results. You can always tell a website that does that shit because it looks like patchwork, it's unacceptable.
I'll give Zepto a shake. I'd never heard of it and it seems to have similar usage at a fraction of the size. I don't give a feck about plugins, so I think it will work great. Thanks!
>>983444
That's a concession for the old tagger I used. You probably won't like the planned alternative though, with the /XX/YY/ folder structure most online boorus have. It's not a desktop app, you aren't really meant to be directly interacting with folders.
If you wanted both, maybe write what I wrote; a script that handles automatically symlinking everything.
>>983542
Any reason GPLv3 might be an issue? TBH I'm old, feeble, and don't understand what you kids and your ball-talk is about.
▶ No.983564>>983565 >>983567 >>983589
>>983552
> Any reason GPLv3 might be an issue? TBH I'm old, feeble, and don't understand what you kids and your ball-talk is about.
Nah, just that real men use GPL, BSD or Apache only. Except when it is Hydrus, which uses WTFPLv2 because he is total savage.
> I'll give Zepto a shake
Good, but could you also use Milligram or Pure.CSS to reduce bloat? Bootstrap and Foundation are both shit (and don't even try bringing up Material Design, most are still bloated)
▶ No.983584>>983588
>>983552
The license is fine. I use GPLv3 myself. You'll always get somebody complaining about the license but as long as you use either MIT or some form of GPL for opensource you are pretty much fine, just depends on your goals really.
▶ No.983588
>>983584
Meant to say BSD, not MIT.
▶ No.983589>>983591
>>983565
>>983564
>>983567
I'm just going to use plain CSS, no frameworks or extras on top.
Also; video thumbnails have a fairly big update coming next push; animated gifs are getting video thumbnails, the thumbnailer is smarter about efficient formats for a slight speed improvement, and it will now properly detect if FFMPEG returned an error while processing.
▶ No.983591>>983593
>>983589
Well I hope that it look sleeker than Danbooru or Gelbooru, on the level of Skeleton CSS (without the importing)
Also be sure to check the issues regularly
▶ No.983593
>>983591
I did not expect to already have reports at this point.
▶ No.983604>>983623
>>983599
I'M GOING TO BEAT IT UNTIL IT'S LEARNED IT'S LESSON.
▶ No.983623
▶ No.983919
>>980927
>one person
It's a whole community you dumb faggot. There was a deep web forum about it for a while that was on the hidden wiki, had all sorts of that stuff. There probably are more that I don't know about. On 8chan we even have a board dedicated to it, >>>/necrozoo/
▶ No.983962
▶ No.983964>>984062
>>983960
So that's why you made this degenerate porn-bait thread. You glow in the dark.
▶ No.984027
A dick cord, for a PHP PoS depending on other libraries.
▶ No.984038
>>983960
Deleted cloned repo.
▶ No.984062>>984065 >>984221
>>983964
... To talk to people and get feedback on a software project? I'm that nefarious.
▶ No.984065>>984219 >>984221
>>984062
You'd have less complaints if you made an IRC server or Tox/Matrix server. Less privacy issues as well.
▶ No.984219>>984242
>>984065
TBH my preferred methods of communication are in reports and pretty much anything I can track. I'm not a fan of live communication channels, because feedback gets lost very quickly in the disorganized stream.
Someone filed a request for me to put up a discord, so I put one up. Discord keeps a history and seems to have basic organization features, so at least I don't need to babysit it.
If I want to shoot the shit with people, I'll do it somewhere like here. If I want to plan development, I'll push the git page. I strongly recommend the git page.
▶ No.984221
>>984062
>>984065
All major anime hoarding projects are in (((Discord))), can't blame them. when was your last time you went to #anime:matrix.ordoevangelistarum.com?
▶ No.984225>>984587
>>983545
Are those the best options for media hoarding currently available? Or you listed them only as examples?
▶ No.984587>>984973
>>984225
Those are the links for combing Hydrus (web scraper and native booru) and Szurubooru (Web booru).
If you want other options you might want to check out :
https://github.com/Bionus/imgbrd-grabber (with limited features, written in C++)
https://github.com/mikf/gallery-dl (CLI written in Python with more coverage)
▶ No.984956>>984969
Nearly finished the SQL re-porting; massive speed improvements, obviously. Page speeds had quietly snuck up to ~100ms for post viewing as I had improved the accuracy of the recommendations. Now with 1:1 functionality it's usually <10ms to generate the same page, always <100. With this as the database gets bigger performance shouldn't tank either.
There's also been a tonne of updates since I posted last. Overall better browsing, pagination is a thing, video thumbnails (and their generation) have been significantly improved, tonnes of cleanup, and widget standardization so things like lists and the tile view are more consistent and featureful.
The big downside is that the depreciated tagger is now totally dead. I had some slipshod scripts do the conversion, but once the port is done I'll need to write a proper scraper setup. I also need to get users up-and-running so these tools can be walled off by admins.
▶ No.984969>>985087
>>984242
His use of "..." is even worse. Vile normalfaggotry.
>>984956
What's with the "?" in front of tags? Do you really have (or want) a tag wiki? I don't think many people actually use it.
▶ No.984973>>985004
>>984587
Thank you, I'll try them both.
▶ No.985004
>>984973
And compare them with Hydrus as it covers more sites.
▶ No.985043>>985048
>>980625 (OP)
I see you didn't blacklist degenerate tags like trannies and co.
This means you will commit suicide in the next 10-15 years and therefore this project will not have a long term maintainer.
▶ No.985087
>>984969
It will have a tag wiki, though it has ancillary functions beyond that. The big thing is that it will let you see how the tag is grouped, aliases, implications, etc. I might also look into adding in some fun stats to the page.
▶ No.985103
>>985048
Damn I love Richard Spencer (ho nomo).
▶ No.985154>>985166 >>985183 >>985185 >>986662
Search is coming along with the database now. I've been fighting between Soundex and Metaphone-based spellchecking. I've got it using metaphone, but I might flip-flop. I also want to make it take tag commonality into account when "smartly" selecting a suggestion. It will also just drop tags not found, letting you know it was dropped.
I still need to add LIKE functionality and the or pipes, but the bare basics are getting together.
▶ No.985166
>>985154
>those results
Are you doing this on furpose?
▶ No.985183
▶ No.985185>>985186
>>980625 (OP)
>>985154
>>981020
good taste
Is this gonna be selfhostable?
▶ No.985186
▶ No.985227
▶ No.985436
>>981811
Hydron dev here. The CLI interface is the one that is complete enough for use. The GUI and HTTP API still need plenty of work.
▶ No.985950>>986021 >>986427
>>980625 (OP)
Hey op you really should just stop downloading lewd images and ask Christ for forgiveness. I was going to share my similiar story here as well as some software I wrote to fix my mess I made, but from your posts like >>983960
I can see you are not only a faggot but also a sexual deviant small brain clown
pull yourself together cum worshiper
SEEK HELP
▶ No.986021
>>985950
You made my morning; thanks for the laugh. Share your software though.
>>Back on topic
I've committed the *almost* database-driven code. It still needs a lot more work before it's installable, but all the parts are there and most of the work is done. I just pushed what I had a little early because the changes were piling up and I wanted to back it up. On the todos are to finish the installer and adjust the config system so it loads data from the database when available.
There's also the fledgling seed of the ajax API. Currently it's used by the installer when probing database connectivity, but it will go much further very quickly (Don't worry, the installer blocks out its own API after installation). The system was already designed, but was stripped down a bit. It should be very robust, and it's only about a dozen lines of code for each new API, if that. In the future I might integrate API keys.
The tag search engine is getting decent. Removing a tag from the search just werks, even if it was a previously mistyped tag that was autocorrected. There's a whole little class for query string interpretation, so I expect that will improve over time.
Images now also have coloured borders depending on status. Right now it's a bit overwhelmingly teal because everything is 'new' in my database, but that won't be an issue for much longer.
... And a bunch of other things.
▶ No.986427>>986483
>>985950
If you can't use it to download porn effeceintly, you can't use it to download redpills from /pol/ threads either.
t. Hydrus user who use it to download 8chan redpill threads
▶ No.986483>>986497
>>986427
just use curl and awk to get links and wget to download em.
You can tie it all together in a very simple bash script
▶ No.986497
>>986483
And that is where things go really wrong, when the amount of files exceeds 10K it starts to take a toll in productivity, especially when trying to find the right resources to post/share.
▶ No.987730>>987857 >>987862
Don't know if people are still at all interested in this; if so let me know and I'll keep going. Lots of updates though.
> The installer is basically complete. I haven't finished the "complete" page, but it werks.
> Users and sessions are a thing. Sessions seems like they'll be secure.
> Page caching en route. Caching is going to be "smart" by monitoring models and clearing pages with affected data.
> The system has an API system now, writing APIs is now trivial.
> Session-level data is kept to the ajax API, being logged in doesn't break use of the cache.
> Moved from jQuery to Zepto.
> Switched to Soundex, which was meant for my use-case (need inaccuracy)
> 'Albums' and fullscreen slideshows are en-route.
> More app options, security options, etc.
> Cleanup & bugfixes
A big todo is to start working on icons, graphics, and consistent visual standards for the project. I've been getting away with some simple glyphs, but it's time to work on 'real' graphics. I've also switched and am working from a safe-for-work imageset now in a second installation. Future screenshots will be, well, safe for work.
▶ No.987857>>987924
>>987730
Please if you could, adopt >>986662 as the algorithm of choice (rewritten of course to make it faster).
▶ No.987862>>987863 >>987924
>>987730
Considering this is basically the only actual technical project on /tech/, I'd say you ought to keep going anon.
▶ No.987863>>987924 >>988026
>>987862
No, this is not the first >>>/hydrus/
But it will not be the last >>>/gnosticwarfare/
Many has been inactive >>>/machinecult/
But you will know when you hit jacktop >>>/agdg/
▶ No.987924>>987930 >>988026
>>987857
The goal of the metaphone/soundex is to get possible tags the user might have intended to type. Soundex was designed to give intentionally broad results, which is ideal because with those results I'm going to be using more computationally intense Levenstien and weighting to select the best of the broad. Right now it's just Levenstien, but weights will come soon. I also need to adjust the Levenstien add/remove/replace values. If I was using *only* one algorithm I'd use metaphone, but since this is a two-stage selection broad is better for the initial lookup.
>>987862
>>987863
Maybe if this project keeps ticking along, when it's more usable it might be an idea to make a board, a la hydrus. Dunno.
▶ No.987930
▶ No.988026
>>987863
I knew of hydrus but I thought its origin was another board. Never saw the other things though, thanks for the links.
>>987924
Might not be a bad idea.
▶ No.988068>>988070 >>988071
Getting started on icons, this will probably be the general style.
▶ No.988070
▶ No.988071>>988093
▶ No.988075>>988076
How about you go chat about your imageboard on your imageboard.
▶ No.988076
>>988075
Boorus are not imageboards (BANI)
▶ No.988093>>988113
>>988071
Goanna make them from scratch; I'm probably also going to replace the glyphs I've been using with custom ones from the KDE desktop. You just get more consistency that way. There's probably also going to be a lot of booru-specific graphics I'll need, so I don't want to either shoehorn in an icon that "kind of" works, mix sets, or imitate another set. Just have a set style from the get-go. Eventually it might also let me do cool crazy things, like apply CSS to the SVG so the icons follow whatever custom theming might be wanted in the future.
▶ No.988113>>988142 >>988636
>>988093
Matrial Design is a good standard though
▶ No.988142>>988203 >>988206 >>988636
>>988113
MD is banal, childish and just outright disgusting. I would never use a device with a UI based around that plebeian UI standard, or else I would feel like I am using a children's toy.
▶ No.988203>>988240 >>988636
>>988142
Suggest a good alternative design principle.
▶ No.988206>>988450 >>988636 >>988941
▶ No.988240>>988450
▶ No.988450
>>988240
That ain't web design, see >>988206
▶ No.988503>>988512 >>988584 >>988604 >>988636
The current boorus are built based on the basic task (cataloging and serving images based on tags) rather than their actual purpose (serving porn). If you are to build your own from scratch, it would be smart to go in with the purpose in mind.
As an example, there are a few different ways in which porn is created/distributed:
>image sets (essentially a primitive animation of sorts)
>cg sets/sprites (VNs, sometimes 100s of almost identical images)
>wips
>doujins
>alternate versions (black/white/cum/clean etc)
As such, it would make the most sense to focus on creating pages based on master versions of images with all related images being hosted under it, rather than having a separate page for every image. This would actually provide a better browsing experience than anything available right now.
I think it's also important to put bigger emphasis on the creators of the images as that's the single most effective way to find more content you like. I don't think it makes sense to categorize draw_fag in the same way as big_anime_tiddies. You could also add social info on those profiles to make it easier to check up on artists you like etc. If artists get pissy about hosting their pictures, you may also be able to appease them by linking to their patreon or something.
▶ No.988512>>988584 >>988636
>>988503
So like exhentai?
▶ No.988584
>>988512
>>988503
IPFS or Beaker/Dat + Hydrus + Szurubooru or Danbooru/Gelbooru would be awesome
▶ No.988604
>>988503
Proper handling of translation sets would be great too. There's no reason why there needs to be a dozen different doujin indexes for the same thing with different translations. Consolidate them under one page and on results show me my preferred language if it's available or the original if that's not available. On the doujin's page show available translations that I can swap to if I'd prefer something else.
▶ No.988636
>>988503
Borehole is definitely being geared towards adult content; not just browsing it but also viewing it. In the screenshots I posted with the "you may also like", I'm also going to be including things like sets, variants, etc. I'm not sure how fine-grained it will be, but I have that under 'core features to target'.
>>988512
I've never actually gotten into exhentai, so I don't know what it has going for it. Frankly I don't really care to either. It takes a whole lot to drive me into getting an account *anywhere*, nonetheless a porn site, with a waiting time, behind a wall I can't see over, when the internet is already a fruitful garden.
>>988113
>>988142
>>988203
>>988206
I think Material is pretty good for phones, but it's less so for applications and the web. The moment Chrome went material design I think its usability took a nosedive. If I didn't need it for testing it would be uninstalled.
With Borehole the 'look' isn't going to change much, nor am I going with some predefined kit. I will be doing a pass to improve consistency, convert some disparate elements into standard elements, etc. Everything will be original, consistent, and fit for purpose.
So, I guess updates;
> Switched back to jQuery. Zepto is missing a *lot* of functions, and I wasn't satisfied with the alternatives, either in libraries or using my own polyfills. I tried, but it was just bloating my own code. So suck it up and download the 60kb difference.
> Some JS code has been streamlined using jQuery-exclusive functions.
> Converted the site over to a purpose-built glyph font.
> Moar icons! Only one is use!
> Transitions have been added. Transitions are also optional, with a site-wide default in the config, and a user-level toggle.
> There is now a preview drawer, similar to Google images. It serves a larger preview, shows tags, and some meta. It's toggleable in code, and will soon be classified as a view option.
> Async operations now have a built-in busy "wave" animation.
> Background operations are kinda working but I'm not 100% happy with them. Scrapers are the main use-case here, but video process will also eventually be done in background. Requires shell_exec, not tested under Windows.
> The beginnings of metatags and functions in search. The options of what will be different than what is available in other booru boards based on performance concerns and usefulness. Functions (like order) will not be OR-able.
> If you were stupid enough to get an installation working, consider it invalid due to database changes.
▶ No.988639>>988651
>>986666
nice article satan
▶ No.988651
>>988639
Say what you want about the guy, but ever since people kept calling sim "Stan" he really pushed the field of phonetics.
▶ No.988653>>988666 >>988696
>>980625 (OP)
kys degenerate.
▶ No.988666>>988696
>>988653
Oh damn, I nearly killed myself that time. If you had of typed the whole sentiment I would have done it for sure! Now I'm all immunized and shit.
▶ No.988696>>988941
>>988653
>>988666
subsatan gets cucked by supersatan6
▶ No.988941
>>988696
SuperSatan here saying that following one of the design principle of >>988206 would be optimal. Matrial Design is just a "default" because Bootstrap and Foundations sucks balls. Pure.CSS and Milligram is the shit
▶ No.989165>>989177 >>989589 >>989600
Update time! Big update.
I've dropped support for a few browsers;
> Internet Explorer (all versions)
> Edge (15 or lower)
> Firefox/Gecko-based browsers (30 or lower)
> Chrome (48 or lower)
> Opera Mini
> Blackberry
> IE Mobile
This is because I re-wrote the CSS (all of it) using some bleeding edge standards, namely CSS variables. In exchange I got to drop a lot of SVG content, and now everything could (theoretically) be recolored by editing one file. It also lets us have some swanky styling that would otherwise be impossible without a fatty framework. I ditched a lot of excess files, and reduced per-page kludges to pretty much nothing.
Other stuff got love too;
>In development mode there's a query debug list for making sure things on the SQL side are nice and quick. Already caught one bad query.
> The footer links are structured out; granted they're mostly dummies.
> Many of the models have been fleshed out. User types/groups are ready, but there's almost no permissions yet.
> Little things, like users having glyphs beside their names when they are part of special groups.
> There's now a default robot named "Boring". "Userless" content will be attributed to the robot, unless otherwise specified. Other robot users can be created, and scrapers can be set to post as those robots. "Posted by SafeBooruBot" kinda stuff.
> The internal user API is in full-swing; log in, log out, log in again from one page. Cached pages seemlessly display the correct signed in user. There is no "pop-in" issues where you see dummy text before the real stuff loads in when navigating.
> Other user stuff like avatars have been done.
> Loads of file cleanup.
> Sound and length as generic post attributes, which will be accessible via metatags. Gifs were a motherfucker for length detection, and aren't guaranteed to be 100% accurate.
I have a couple experimental things I haven't moved into the trunk yet.
> An optional canary. Any administrator can enter their password to reset the canary; if the canary expires, a warning telling users so will appear, also explaining what site canaries are. Once enabled it can only be permanently disabled with the root administrator password. I might research how these are made in case my implementation has a loophole.
> A stripped down version of Activerecord. I really don't trust it, but it's unmaintained code so I want less of it.
For users I've put in a configurable Karma system into the groups system. When users have permission to do something they will probably have daily limits set at a base value; the karma system will add to or remove from that limit. Karma could also be used as a carrot and stick for admins/mods. Karma is earned through up/downvotes in comments and posts. Additionally, user types can be configured to require karma to enable certain actions. There's no admin panel for it yet, but I've made several nice defaults to bundle on new installs.
The big goal right now is to finish up what I started, clean up what I've got, and polish rough edges. Ideally I want the system to be easily installable for testing purposes before next weekend so people can see what the hell this is all about.
▶ No.989172>>989389
▶ No.989176>>989177 >>989389
>>980625 (OP)
BETTER WAYS TO RATE IMAGES AND DISTRIBUTED NETWORKING
▶ No.989177
>>989165
>>989176
ALSO FILTER THE SITE INTO ONE GIGANTIC OVERBOARD WITH DEFINED SUB-BOARDS
FEATURES FOR "SET"S OF IMAGES
VISUALIZATION OF ASSOCIATED IMAGES (WEB GRAPH FORMAT, MAYBE?)
▶ No.989246>>989251 >>989279 >>989389 >>989605 >>989609
Integrate XPcoin blockchian tech, whenever you upload you get xp to encourage content, add incentive for completing certain tasks.
fund site, through tax using xp as purchases for collection downloads.
https://www.xpcoin.io/
▶ No.989251
▶ No.989279>>989295
>>989246
I almost fucking barfed when I read the xpcoin website. It's like trying to control ants with sugar. What a disgusting idea.. not much unlike modern capitalistic society though. Can haz plz freethinkers.
▶ No.989295
>>989279
i normally don't do this but >>>\leftypol\ Fuck off
▶ No.989389>>989390 >>989517 >>989605
>>989172
I'm not planning on making a carbon-copy API; but it will be roughly equivalent. Part of this is because my data structure is a little different, part is because the danbooru output can be improved slightly (why are tags returned as strings?)
It's very likely I'm not going to support XML. JSON is just lighter, more compact, understandable, and easier to parse without errors. I actually started with XML support, I'm just removing it.
>>989176
Hello Red. I'll get into my evil plans for the rating systems; I think I'll be implementing two.
The first will be a 4-star rating system. Super simple for users. For API compatibility it will translate them into up/downvotes. I'll also do a little bit of math with votes to improve their weighting for the actual star value; the further a voter deviates from their norm, the more impact the vote will have. E.g. a user who 4-stars everything will have an impact of ~ 0.6 votes, but if they 1-star something it would have a ~1.4 vote impact. Weighting will be system configurable/disable-able.
For the second half of the rating system I'll implement reaction voting. Users get a set of reaction emoji (love, meh, lust, dislike, barf, awe, etc - tbd) and may choose one.
The fun is when you combine the two. In the database the star rating and the reaction will be in one table. Outside of sorting by raw rating, sort by how awe inspiring an image is. Heck, for users into that stuff you could sort by how (positively) sick an image is.
That's the plan, at least.
On distributed networking, I've had it suggested before to use include IPFS. I personally haven't worked with any of these things, so my hope there is someone who knows about that stuff might contribute the code. For my personal manpower I want to get the basics running before this turns into technological masturbation.
>>989246
The main downside to stuff like this is the discouragement of genuine passion, and the introduction of a system with value to be manipulated. It also goes directly against the Karma system I'm developing, where positive community interaction gives the rewards.
▶ No.989390>>989517
>>989389
Correction; Why are tags returned as a single string? Also, it limits how well tags can be described. Etc.
▶ No.989517>>989780
>>989389
>>989390
Make standard adapters for backwards compatibility would make this better than Danbooru, Gelbooru, Moebooru, Shimme and MyImouto (the top 5 booru engines with API)
▶ No.989585>>989780
Hey OP, you could
put this on your CV and get a real job faggot
▶ No.989589>>989780
>>989165
>I prefer CSS bloat over browser support
D
R
O
P
P
E
D
▶ No.989600>>989780
>>989165
> Not using a standard CSS package like PureCSS or Milligram
Boo
▶ No.989605
>>989246
>>989389
steem.io > (((XPCoin)))
Bitchute, Steemit, what's next?
▶ No.989609
>>989246
>used to have a team page that's full of pseudonyms
>but looking into their kikehub they have Japanese and Western devs
>has a significant Japanese community
<muh /v/ and eSports streaming: the shitcoin
Shill somewhere else faggot
▶ No.989780>>989877 >>989879
>>989517
I've just sketched in rough plugin support; I don't have anything that would affect existing pages, but it's relatively simple now to extend the configurations and add new controllers. Over time I imagine it will become more robust, but right now it's the bare minimum to add logic.
But what I have is enough to add a skeleton API example, so now adding APIs will be childs-play, and you don't even need to mess with the actual core system. You just set a route for your API, and point it to the controller which handles the logic. It eager loads plugins now, but it'll be trivial to add toggles later to enable/disable individual plugins.
>>989585
My CV is already baller, it tends to happen when you actually have work ethic.
>>989589
>>989600
If you use Internet Explorer or a browser several versions out-of-date, then you have bad taste in software and I'm the one dropping you.
Bloat-wise, you need to understand that adding things is bloat. There's no magic there. Frameworks aren't packaged fairy-dust, Tinkerbell isn't flying around behind the code magically doing voodoo to make things lean. They're just somebodies idea of a good standard design, and yeah, some of them are lean for what they provide. The problem is that they all provide generic shit that needs another layer of styling and workarounds. That's not including the parts you aren't using bundled in.
If you code pure CSS efficiently it's exactly the same, only the site doesn't look like white minimalist flour or need a heap of adjustment/spackle CSS.
▶ No.989877>>989935
>>989780
But Milligram's "bloat" is small and Milligram is even smaller, custom-baked CSS just don't look good when compared with ones that have proper design principles in mind.
▶ No.989879>>989935
▶ No.989935>>989938
>>989877
> Assumes I don't know design principles.
>>989879
Ah! I can see how that would freak people out. No worries though. Calm, calm, calm blue ocean. There's a big difference between a browser and a client. Those are clients, and are getting 110% love. They work via APIs (I confirmed this with a few of the devs and also asked about what APIs to target first, just for you guys) and they don't go anywhere near the HTML - which could still be scraped.
I'm talking about browsers;
For mobile users browsers using cloud compression, which pass all of your data through proprietary servers to serve 'optimized' pages, aren't supported. The ones known for collecting literally all of your data, which is tied to your phone. Even then, with apps like Opera Mini, support will come when the developers update their compression server software. So if you want some company to know the porn you've been browsing, you will, eventually, be in luck.
For desktop users, when I made the snark about using IE I was actually being pretty literal. No browser on the market today is incompatible if you've updated it in the past year. Even Tor, which uses notoriously outdated versions of Firefox, is compatible.
▶ No.989938>>989947
>>989935
I am saying that those mobile clients needs standard APIs to view any image gallery server including yours, and nothing being able to use a faplet to view your own stuff is just SAD.
▶ No.989947
>>989938
It's going to be running standard APIs through the plugin system. I already have a skeleton Danbooru API in there (which in part I'll be using as a sample API), as the project reaches feature-parity with existing boards the API will cover the standards.
▶ No.989949>>990125
Update before sleep;
The installer is finally finished up, barring one minor developer-centric feature, and one security patch. It is now possible to fully install Borehole in <5 minutes on capable server. You will install an empty image board with an empty admin panel and no way to upload content - but damn if actually doing the installation isn't comfy.
▶ No.989990
▶ No.990125
>>989949
Just remember that we appreciate you anon
▶ No.990259>>990277
Cool update; the recommendation algorithm has been improved.
So a major problem I had with the recommendation engine has to do with the concepts of 'bias', 'qualities', and the number of tags to work with.
On "bias"; the algorithm takes more common tags and lowers their "weight" when finding and ranking posts. It makes sense on paper; a broad tag like "female" isn't as important as a tag like "red_stockings". When deciding what to pull from the database we can't process 30,000 posts because they matched "female", we need more focused results. This made the algorithm very biased towards the 'rarest' tags, usually characters and artists. At first, I thought this was fine, but in the back of my mind I knew something was off.
Then "qualities" became validated my concern; when the system focused too heavily on rare tags, it would regularly forget the "big picture", the qualities of of an image. If you looked up sexy lady Zelda, it would give you Zelda, but you didn't necessarily want Rule 63's Zelda swinging a wang in your face too. You want sexy lady Zelda, or even sexy ladies like Zelda. So to remedy this I made the algorithm more "impartial" to tag weights.
The problem comes if a post has fewer tags is scored impartially; it totally defeats the weighting system. The impartial results will latch onto anything like someone drowning, and result quality takes a nosedive. When scored with bias, posts with lots of tags cling to only a few rockstar tags, discarding the many qualities tagged in the image. We need to have our cake and eat it, too.
The solution was to make the recommendation engine use adaptive bias, and use fuzzy-logic. If a post just doesn't have many tags to work with, the engine will take those rockstar tags and leverage them. As posts get more tags the engine mathematically tweens towards impartial weights to offer results based on the image as a whole. The result? Posts with a few tags get good results focusing on their strongest elements, posts with a lot of tags use all their qualities, and everything between is a balanced blend between the two extremes.
Which brings me to the next problem; complication. Now the recommendation engine is making freaking decisions, using fuzzy logic, and in the transition to the SQL-driven database I stripped out the debugging layer. This one was simple though; I changed up to use a parameter bag, and then added a feedback reference parameter. Now not only is there fine-grain tuning for the logic, but there is once again high-quality feedback letting us know what the recommendation engine is doing.
On a side note; a lot of this work has been done on kickass long vacation. Vacations end. I have another couple weeks, but soon this will bein to follow the expected pace of s normal side project. I hope to get something worth installing before then, even as a toy, but I figured I'd let'chall know that this pace won't last forever.
▶ No.990277>>990540 >>991122
>>990259
Give some finer details on the search algorithm, like the idea and would like to implement that for >>>/hydrus/
▶ No.990540>>990636
>>990277
I wonder; maybe I should open up a blog for this. What's actually decent nowadays?
As a matter of fact, anyone have recommendations for a good host/domain registrar? Ideally one that I could use in the future to run an r34-type installation without constant threat of takedowns.
▶ No.990636>>990682 >>990711
>>990540
Medium is shit, >>>/hydrus/ uses Tumblr, steemit/Tavrngg/minds/gab all sounds good
▶ No.990682>>990696 >>990711
>>990636
>recommending tumblr, shillit and sites that don't function at all without JS
▶ No.990696>>990711
>>990682
Then self host it faggot, not like I have to help you or anything. Neocities anyone?
▶ No.990711
>>990696
>>990682
OP here, that wasn't me. As a rule of thumb I won't insult ideas I decline - unless I was jabbed first. I'm a happy soft nice-guy marshmallow.
>>990682
With HTML5 Javascript is now considered a part of the standard; interestingly before HTML5 sites "should" have had fallbacks, but now the working group says disabling it or not supporting it is breaking the spec.
Borehole requires Javascript; at least if you want to have an account. That being said, I'm trying to keep it usable without it for anon users.
>>990636
I probably won't use Tumblr, they've been purging a lot of blogs lately (a couple I used to keep bookmarked were taken down); I'll check out the others you mentioned. Thanks!
▶ No.991122
>>990277
I was planning on doing it in a blog post, but Gab wasn't as robust as I had hoped. So I took a bit of time, polished it up, and commented the hell out of the recommendation algorithm.
https://gitgud.io/Dizmal/borehole/blob/master/borehole/models/post.model.php
The function you want is "getSimilar".
There are a few math functions in there, the ones to pay attention to are tween, sqroot, and easeOut. Tween just does what's on the can; it finds the between value of two values. SqRoot is a square rooting function which accepts the number of times to perform rooting operations (including fractional rooting). EaseOut is essentially just Tween, but puts the output on a curve. Anyone who knows animation or game maths will know exactly what's going on there.
In terms of using the inverse square root, I have yet to implement that into my sqRoot function. Before I commit something like that, I'm going to be doing a benchmark to see if PHP or native is faster and I'll let you guys know the results.
▶ No.992203>>992248 >>992249 >>992344 >>992346
This is the third reply in a row by myself; Dunno if it's just because the last updates weren't interesting, or because interest is waning. If it's the latter I'll continue development, but move bloggy stuff to a proper blog somewhere. There was also a bit of a delay because I had a drive crash and had to reinstall my shit.
But yeah! updates!
> Improved installer, thanks to a clean-slate server I moved feature validation to the welcome page (before rewrite is required)
> Plugins are now a real thing. They can bind to triggers, extend context, redirect the class loader, and own contexts. They are still eager loaded.
> The tag scraper is in early testing. Currently the e621 can (very inefficiently) scrape tags. Soon it will go from n*n queries to 3 per batch. This affects all scrapers, and there's also Danboory and Safebooru scrapers which just need minor work to work.
> What I think are the last changes to the recommendation engine for the foreseeable future. Pretty much just bug-fixes.
> Recommendation feedback is now displayed pretty and inline with results - but only in dev mode.
> Config can now be driven by the database in tandem with config files. Any config setting in a config file can be overwritten, such as site title, number of posts per page, etc. There is also now an optional site tagline to quickly describe a booru.
> The logo has been replaced with a nicer version.
> Moar icons & glyphs
> Nicer error pages
> The search bar can now be moved; options include the homepage, sidebar, top navigation, and above the content.
> There is now "pretty" tags in the search bar. Eventually this will add functionality, but right now they're just visual. There is single-line mode and multi-line mode (for the sidebar)
> Pages now have unique titles.
> Gitignore to make things easy.
> Admin panel taking shape.
> Tag names on display are now titlecase, as opposed to underscored.
> ... And more!
It's damn near ready for basic testing installations. In order to continue I actually need to add in the last code required to fully bootstrap the database when there's existing content (due to the drive loss) so it's been a real incentive for me to finish up that part.
▶ No.992248>>992249
>>992203
Please make your tag scraper and downloader compatible with Hydrus' so things can be easier
▶ No.992249
▶ No.992344>>992606
>>992203
Looking forward to the tag scraper so I can finally ditch (((hydrus)))
▶ No.992346
>>992203
Keep up the good work.
▶ No.992606>>992645
>>992344
> Throwing all the downloader scripts out the window
> Removing the whole PTR
STFU (((Pyrus))) (See >>>/hydrus/10128 and >>>/hydrus/10129 for his faggotry)
I would rather see a Wx => PySide2 transpiler with modularity for multi-computer proxies and IPFS.
And also a WebUI for browsers, or an API for mobile. Maybe Fuzzy searching and tag correlation statistics.
▶ No.992645>>992649 >>992652 >>992886 >>1001215
>>992606
In the past I've found the hard way that making a feature list first and building that list are two very different beasts. I used to think I was a goddamned wizard because I could point out flaws. After a few years I found out how f'ckin easy that is. Getting to it and writing the code? Damn near impossible. Hell, even as far as Borehole is getting, the odds are against it. Even if you do it 100% you're still really standing on the shoulders and experience of your predecessor projects.
It just pays to have a respectful attitude in general to people who wrote whatever software. They built the concepts, they've heard feedback from users, they know what's really important, they've dealt with the 'gotchas' that come with the software. Why would you give them any reason to say "fuck off" when a modicum of humility will get you in with peers holding that knowledge. You only have any insight at all because they took the idea you like far enough.
I have criticisms of literally every imageboard. Theirs are criticized. Mine will be criticized. Future software - wait for it - will be criticized! But You know which projects I'm completely unable to criticize? The ones that don't exist. It's a fucking testament that the software exists and is known enough to even to warrant criticism.
Pyrus? Dunno if you're the one reading this, let me tell you as a fellow developer; our projects? They're hatchlings, ugly-ass featherless bind hatchlings. Hydrus? Danbooru? They're the 10-pound roosters who scrapped in the ring and came out alive. Talk is cheap, write the code, know anything you do is thanks to the road they paved for your ass.
▶ No.992649>>992652
>>992645
I'm sleep deprived. Maybe a little drunk. Grammar is in the shit. But dammit you get the idea!
▶ No.992652
>>992649
>>992645
Dev is humble, nice!
▶ No.992886
>>992645
My only criticism of this post is that hydrus-chicken should be ill-proportioned, retarded, and missing some feathers, with error messages across its stumpy flightless wings. Flight Support Coming Soon™. It does however have scrapbook with a million pictures of cockerels. :^)
t. hydrus_dev
Keep pushing, it is worth it. Send me a mail if you would like advice or help with anything. Not that I have much technical skill to offer, but as you say, I do have some experience now.
▶ No.993105
Alright, newest update in the works has some neat stuff. This update isn't about what I've added, but what I've taken out. A little more work is due before I upload the final stuff, but I'm hitting the sack.
> Massively simplified the View system. Before it was two classes, had all sorts of recursion, parent/child relationships, and loads of over-engineered stuff. It's all gone. Now it's one class, sequential, and much more flexible (ironic!)
> Consolidated some similar systems into generic systems, then tacked on a few convenience functions. Things like the CSS loading system (which was technically broken!) now just use a generic system in the new view class.
> Fixed some issues with the config system, and made it so it can accept mixed associative/indexed arrays. This simplified the site navigation config.
> Removed a lot of "Borehole" specific links and utilities then moved it all into a plugin. Now a site admin could disable it to remove my precious links to things like Discord. The plugin also detects the .git folder and displays the build hash when enabled.
> Removed lots of "placeholder" links and content. The site is increasingly config driven, which is good because it also extends the plugin system.
> Moved the canary code into a plugin. I'm currently using it as the first plugin system stress-test, and it's what revealed the config issues.
> Themes are evolving. The default theme is "Drill Bit" and is the purple theme. I'll also be making a theme named "Boring" which will emulate existing blue-on-white Boorus, and serve as a skeleton theme. Themes are currently just CSS and a manifest file, but soon they will also provide overrides for template files.
> In addition to themes, the counters can be skinned independently. Currently I have the "flipper" style (the default) and "counter-girls" which is those catgirl gifs. Counter skins are just CSS, images, and a manifest file. The catgirl gifs use pixelated up-scaling because as fuzzy as catgirls can be, blurry catgirls are no fun. Note: for me this is the first time in eons Firefox was the culprit of a namespaced hack.
▶ No.993488>>1001288 >>1001391
Themes in action. Themes are broken into 4 parts: The skin, the colour scheme, the icon set, and the homepage counter.
▶ No.1000590>>1001174
▶ No.1000634
▶ No.1001174
>>1000590
Nobody gives a damn.
>>>/hydrus/
▶ No.1001215
▶ No.1001288>>1002777
>>993488
you're a bro for adding themes but the dark isn't dark enough tbh
▶ No.1001391>>1002777
>>993488
Any feature updates on this? You should keep a changelog.
▶ No.1002777>>1003445
>>1001288
It's not really meant to be a dark theme, just happens to be dark(ish). That being said, the colors css file is obscenely easy to modify.
>>1001391
Sure, I've been avoiding posting in this thread because I didn't want to spam my project if interest dropped. There's been quite a few since the last time I posted here, some highlights include (and sorry for dups if I've already said some of this stuff);
> Plugins are now manageable in the admin tools.
> Plugins have more hooks and more reliable ways to extend site configuration.
> Plugins include: "Borehole Extras", "Site Canary", "Debugging Tools", E621 Scraper, Danbooru Scraper (dummy), and Safebooru Scraper (dummy); Borehole extras is some links & info I'd *like* sites to show guests (links back to things like the gitgud page) but don't want to force sites to show. Site admins can remove my extras without forking or modding code. Debugging tools depreciates the need for debug/production databases, allowing devs and admins to kick a site into debug mode at will.
> New config tuner. Similar to about:config in browsers. Certain values (such as the db connection string and root user ID) are blocked from the web UI. Mainly meant to be an advanced tool for corner-cases, but useful during development too.
> Math library with a string calculation tool. E.g. the string "2 + (4 * 3)" will output 14. Doesn't use eval, but a custom parser. Used in the config tuner for numeric values so you can enter stuff like "60 * 60 * 24 * 7" if you don't remember the result off the top of your head.
> Beginnings of user posting in. Supports drag & drop and file selector. Not yet functional, but does 90% of the work needed.
> Re-wrote background processes; background processes can now be tracked in database and rogue processes can be found for killing - if they ever happen.
> Processes may report the number of completed/total operations for "live" progress tracking.
> New migration & import tool will organize images into desired folder structures, generate thumbnails, and optionally auto-tag images using enabled scrapers. Runs in background and reports progress.
> Posts default to the ab/xy/md5 folder structure - flat posts are also accepted. Posts don't require MD5 names, but it will automatically sort them into md5 names and the primary admin-defined folder structure if requested on import.
> Sound is now part of the autodetected built-in metadata. Thumbnails now show audio icons if an audio channel is found in the video.
> New algorithm extracts color palettes from images. Each post has a primary, secondary, and background colors associated with them. Used in thumbnails. Will be used for style enhancements and flourishes later.
> The system now goes by metadata for most operations vs going by extension.
> Overhaul multimedia and image processing library in the works. Nicer, more efficient code using variable bags.
> Lots of stuff in general has been overhauled and polished. Bugfixes. Code churn. Etc etc.
There's been a tonne more, but these are the highlights I think. I've been horrible about keeping a proper changelog, mostly because this software isn't "ready" yet so I don't see a point. I've started putting change lists into the git commits, and will start keeping proper changelogs when it's ready for even the most basic production usage. At this point though it's still a bit too early, especially with fairly regular breakages.
▶ No.1002879
Screenshot of post colors in action.
▶ No.1002882
Oh, and another with the content the colors were pulled from.
▶ No.1003445
>>1002777
Trips, now make an API.
▶ No.1004769>>1004770 >>1007002
>>980655
As long as the art is good and there's separate categories/filters for anime and western.
▶ No.1004774>>1007002 >>1007226
>>1004770
Make that separate categories for furry and kemono too. Easy to filter and look for specific stuff.
▶ No.1004820>>1007002
is there a live instance of this anywhere?
▶ No.1007002>>1007375
Turns out everyone turned into a blithering moron while I was on vacation, and kindly piled up a massive amount of overtime for me. Literally haven't had a day off since I stepped back into the office.
>>1004820
Not yet. I'm looking for a good hosting provider that is anonymous, overseas, and doesn't shut down clients at the drop of a hat. I have a couple earmarked, but if anyone knows a good hosting provider please post here.
The plan, at least for now, is to host a project website /w dev blog and sfw sample site. Once I ensure security is tight on the admin side I want to host a proper rule34 site.
>>1004769
>>1004770
>>1004774
I think this fits into something I was already doing. Tags will have types (like subject, metadata, media, etc). After thinking about it I decided to make these softcoded so site admins can add the tag types they want. Site admins will also be able to make some types *require* a quota of at least a number of tags before the post can be approved. E.g. a post must have 1 author tag and 5 descriptive tags.
Borehole is also going to have the option of using a separate "new" area for posts not yet approved. In order for posts to get out of new subs they'll need to meet a few requirements, one of which being the aforementioned tag quota. Posts which aren't approved won't appear in tag searches.
Together this probably fulfills your request. Admins just need to make some sort of "topic" or "category" tag type then require at least one tag, and then you'll be able to filter as per normal. E.g. "western -furry".
▶ No.1007226
>>1004774
kemono is furry though
I know e621 has a kemono tag, so it shouldn't be too hard to just use kemono or -kemono depending on which region of furshit you want to jerk it to.
▶ No.1007375>>1007376 >>1007625
>>1007002
For servers, a colleague of mine recommended Contabo, hosted in Germany. Not anonymous, good service and not dropping tor nodes or such, relatively cheap. https://contabo.com/?show=vps
To be honest, you'd be better off hosting yourself and setting up a hidden node.
▶ No.1007376>>1007388 >>1007625
>>1007375
Note that loli is banned in Germany so you can't host that there. Like you said, better off self hosting it over tor with an onion (v3) address if it's just a public test server.
▶ No.1007388>>1007625
>>1007376
Didn't know about that, thanks friend. Is it a Germany only thing or an European thing?
▶ No.1007625>>1007690 >>1007702
>>1007376
>>1007375
>>1007388
I'm not planning on hosting loli or illegal content. As a matter of fact I'm going to be deleting what I have once the features are in place on my local installation; it's just not something I refer to anymore, and I haven't really had the motivation to delete it manually. I'm also interested in doing everything through the web interface as it develops for dogfooding purposes.
The r34 site I want to start will host "everything unreal" except loli and guro. Photos, cosplay, and 'realistic' 3D are going to be out - at least on my installation.
On anonymity, I'm involved in several communities and it's just not something I care to be deeply associated with. Additionally, people love to dig; if I were traceable back to an archive of this thread it could potentially end me. I'm not interested an anonymity for the hosting of illegal content, I'm simply 80% private and 20% paranoid.
▶ No.1007690>>1007694 >>1021722
>>1007625
>The r34 site I want to start will host "everything unreal" except loli and guro
>Photos, cosplay, and 'realistic' 3D are going to be out
>I'm not interested an anonymity for the hosting of illegal content
Sounds like you're confused what "illegal content" is. I mentioned Germany law to him because I assumed (rightly) that he didn't know they outlawed loli, so people can't host it at the company he mentioned. That doesn't mean you can't host it elsewhere. It's perfectly legal in the US and other countries to host it. I suggested hosting it as a hidden service on tor because it's fully anonymous, as in there's no need to register a domain name and rent a VPS. You can self host it in your basement off an old laptop if you have decent internet speeds. There's no personal information you need to give out and no one can cut off your hosting for the content you serve.
Personal preference wise I agree with you blacklisting gore and 3D. But why also loli if you don't have to? Is that also your personal preference? What kind of person is into furry but not fine with loli?
▶ No.1007694
>>1007690
I fucking love MUK. Also I agree. Just host it where they don't care about loli.
▶ No.1007702
>>1007625
For domain name, Njalla seems to be a good option. https://njal.la/
▶ No.1008005
Ok, report back. Changelog?
▶ No.1008368>>1008399 >>1008535 >>1009147
Bump. Get in here op, too long without news. Is Borehole dead? Alive? What's new?
▶ No.1008399
▶ No.1008535
>>1008368
Alive and well, I'm just a bit swamped with work so I've had to stay back at the office to catch up. No real progress since last update, small cleanup and Tony todos, but nothing worth a big write-up for. I've only had the odd hour after dragging my tired ass home, so I haven't had the productive hours on end of a single day off. I'm salivating to get back to it though, just as soon as work calms down again and I get the chance.
▶ No.1008797>>1009161
>>980625 (OP)
Is there a fucking good tool or program to organise millions of images through tags like this fucking bloated slow hard shit called hydrus but simple and fast and work well and doesn't need you to be expert in coding to understand shit ???
▶ No.1009147>>1009399
>>1008368
Just pushed a small update which fixes a major regression, and updated the readme as well. The installer was broken because of an update to the template system, which is now fixed. Testers can now, once again, install Borehole.
Additionally, there are now instructions for importing images. There's no auto tagging, which is what I'm polishing up now to use scraper plugins.
▶ No.1009161>>1010549 >>1014408
▶ No.1009399
▶ No.1010549>>1010694 >>1017589
>>1009161
Hi there hydrusdev!
You still mad some literal who part time go programer managed to make a better tag based system then your bloated barely working python shit?
▶ No.1010694
>>1010549
Bump it if you dare, fag
▶ No.1010714
▶ No.1011119
▶ No.1014408>>1017589
>>1009161
>>hydron
You still need to import shit to it
▶ No.1014691>>1017586
Various furfag shit
>No Libbe
REEEEEEEEE WHERE IS LIBBE ON MY /TECH/ BORN IMAGEBOARD
▶ No.1016569>>1017586 >>1017589
Small update because things have been kinda slow; work has been taking it out of me since I finished my vacation, things are starting to get back to normal, but between overtime and general exhaustion I simply haven't had time to do, well, anything. Even the days off during the holidays were too busy.
Things in the pipe include the new background-process tag scraper, optimizations, a new thumbnail/sizing system, and more control panel utilities. There has been more installer work fixing some nasty issues that popped up, and I'll also be making it more verbose when it hits errors. There's also work on the API, which will be modeled on work by the hydrus devs.
▶ No.1017586
▶ No.1017589
>>1010549
see >>1014408 (bakape a fag for Hydron AND Meguca)
>>1016569
> There's also work on the API, which will be modeled on work by the hydrus devs
Salutations for based devs!
▶ No.1021116>>1022833
any news on the new booru you are making?
▶ No.1021722>>1021731
>>1007690
People don't want to host pedo shit because it's goddamn pedo shit. It doesn't matter how you justify jacking off to children you're still doing it. When you allow loli the people wanting real children start hanging out and trying to upload that sort of thing.
People don't want to visit those sort of websites and many of the userbase these places could have will be driven away by it being disgusting and illegal in their region. Just fuck off to discord or better yet kill yourself you pedo freak.
▶ No.1021731>>1021734 >>1021738
>>1021722
>complaining about loli on an imageboard that allows loli
>discord
Fuck off you retarded goon.
▶ No.1021734>>1021738
>>1021731
>Telling people they're sick for jacking off to children makes me a goon
Fine I'm a goon. At least I'm not a pedo.
▶ No.1021738>>1021741 >>1021755
>>1021734
>>1021731
Both of you are going to the gas for dissing fashy discords
▶ No.1021741>>1021753
>>1021738
Could you be a bigger faggot if you tried? Go back to your gay pool parties.
▶ No.1021753
▶ No.1021755>>1021756
>>1021738
>This platform that has doxed and sold information on wrong-thinkers just happens to let these fascist groups remain it's not a honeypot at all fellow anons, why don't you sign up?
▶ No.1021756
>>1021755
> Forgetting that the best organizations moved from (((Discord))) to Matrix
▶ No.1022051>>1022069 >>1022833
>>980625 (OP)
>So I wrote another script which would scrape various booru sites and get the tags for the images I had, dumping them into a text file.
Do you happen to still have this script? I don't care about maintaining databases or a local imageboard. All I want are tags in my filenames so I can grep my pictures folder.
▶ No.1022069
▶ No.1022833>>1022914
>>1021116
I'll be posting an update sometime tomorrow, along with a link to a demo site when I finish getting it online.
>>1022051
Probably, but tbh you wouldn't want it. It was catered to my exact setup, only for the boorus I went to, and had exactly 0 configuration or options. It still required PHP, and I doubt a whole lot of people could even get the original scripts running. I'd recommend a proper program, I've never used TMSU, but I'd imagine it's better than what I had running.
▶ No.1022900
>>980625 (OP)
must be one hell of a brain disease
▶ No.1022914>>1023029
>>1022833
> TMSU
> Not Hydrus
▶ No.1023029
>>1022914
Mostly because my un-option was command-line and scripted, not GUI.
▶ No.1023309
Pushing my little update back a couple days; I've been committing most of the changes, but the proper changelog will need to wait.
▶ No.1024493
Still working on stuff, but here's some of what's been going on.
- Debug Mode is now a config option instead of something special.
- System errors will now always be printed.
- background processes now use site PHP ini file.
- Some post metadata is now being shown, more/less what I needed for development purposes.
Scrapers are now up-and-running in the background;
- Scrapers no longer reports site URL in agent string (fixes cloudflare block)
- Scrapers are now much more efficient, cutting queries in half and using Dapi when possible.
- Scrapers are now sharing a generic codebase mostly based on Dapi, with extras for non-compliant sites
- Scrapers respect rate-limiting to avoid 504 errors. Rate limiting is tracked per-scraper, running multiple scrapers is only as slow as the slowest individual scraper. Time included processing other scrapers is factored in, so scrapers doesn't wait any longer than needed.
- Rule34.xxx, Danbooru, E621, and Gelbooru now have scraper plugins. Safebooru removed. All scrapers are rate-limited at 1 post/second.
- Automatic scraping is using geometric scheduling per-post. As posts get older and it's less likely a source has added tags, the scraper will run less frequently over time.
Borehole should be usable with test installations now, with limited functionality. Installation, importing images, then auto-tagging should offer a browsable experience. I'm currently prepping a demo installation, but it's running on a potato and I have annoying logistics so it's taking some more time than I hoped.
▶ No.1026241
i actually got through this thread without fappan
nofap is hard