SPC is slow and FSE is fucked and I cannot find the thread we were in and I just got out of meeting and have more meeting so I will be back but here's the thing I was gonna post:
> baest is borked again and not federating with fse
The problem has been that it is federating way too enthusiastically. As of today, SPC is doing it also. (Maybe it was before, but I don't know, the machine crashed almost as soon as I sat at my desk.)
But that gives us something to work with! Per the fedilist data, SPC updated from a835067c to de06c1cd between 2024-01-09 19:36:26 and 2024-01-09 21:39:13. (cc @Moon .)
Whatever has started causing the inadvertent DDoS was added between those two commits. I will check shortly, have a meeting.
@Moon@Terry@colonelj@colonelj@iamtakingiteasy@lanodan@mint@sjw Yeah, basically it's trying to deliver to /inbox, then it tries to deliver to the personal inboxes of, apparently, all of the followers of whoever originated the activity. That cascades: the origin server returns a 4xx or 5xx because it already has that activity, so delivery is retried later.
@p@Terry@colonelj@sjw@iamtakingiteasy@colonelj@lanodan@mint@Moon ok I thought I saw this happening like a week ago but then I thought I was crazy. More investigation needed.
@p@Terry@colonelj@sjw@iamtakingiteasy@colonelj@lanodan@mint@Moon this problem goes away if you have my latest optimistic inbox changes as they'll just accept the post with a 200 and throw it away so there will be no retries :toot:
@p@Terry@colonelj@sjw@iamtakingiteasy@colonelj@lanodan@mint@Moon yes but it lowers your incoming traffic. It's the only thing you can do on your end, so I'd recommend pulling the latest changes.
FSE is down and it is still happening: ping times to the gateway are worse than ping times to Cockbox right now. It is a flood. When FSE is back up, it will still be a flood but it will be hosing my DB. The problem is on the other side. If you are delivering every activity 150 times, it's hosed already.
Returning 200 isn't a fix: it just makes it get delivered 150 times instead of 450 or 750 or whatever the retry threshold is. A fix would be delivering once.
Someone clicks Like once, and that's one request that triggers a delivery to N servers, right? Unavoidable. But now it's one request triggering one request per follower. That is a catastrophe. That is an accidental DDoS already.
@p@Terry@colonelj@sjw@iamtakingiteasy@colonelj@lanodan@mint@Moon if you can't handle this small number of activities I feel very sorry for your hardware. Imagine when the fediverse actually gets big, you're gonna be so fucked lol
> if you can't handle this small number of activities I feel very sorry for your hardware
"Small number of activities" is different from "small number of activities multiplied by the number of people following an account". O(n) is fine, O(nm) is the practical reality of fedi, O(nmo) is out of hand, and we're currently O(nmop*q) for this bug.
If the site is down due to hardware problems and the problem is still happening, then the issue is on the other side of the pipe.
> Imagine when the fediverse actually gets big, you're gonna be so fucked lol
There are 12 accounts on your instance and it's fine: I don't think instances should be too big. But it does mean that you should probably listen when someone mentions a scaling issue. You have had this attitude towards scaling issues since the beginning: you don't observe a problem, fine, but you could stand to listen when someone else does see the problem. FSE's DB is a little over 500GB on-disk, about 400GB of attachments, and it is probably a reasonable idea to listen when I tell you that there's a scaling problem, but you tell me it works on your machine; fine with me, I'd rather look for and fix the bug than argue with you about whether it exists or not.
My approach with the "maybe 100k images in a single directory is a bad idea" bug that you marked as a dup ( https://git.pleroma.social/pleroma/pleroma/-/issues/1513 ) was to just write a better media server, my approach with https://git.pleroma.social/pleroma/pleroma/-/issues/2765 was to write a better server for the /objects/ endpoint, my approach with the activities_visibility index was to send a patch, my approach when you told me that Pleroma could handle the load on every instance and that you've never seen a performance problem was to brush it off with a joke instead of arguing with you.
So please leave me alone while I read through this codebase and fix a bug. I'll send a patch if I come up with one, and everyone's life will be better than it would be if we sat around arguing.
@p@Terry@colonelj@sjw@iamtakingiteasy@colonelj@lanodan@mint@Moon I'm sorry but you're exaggerating everything. This change has not even made it into a Pleroma release yet. The number of servers running this code can probably be counted on one hand.
> This change has not even made it into a Pleroma release yet.
If you mean the priority inbox change, doesn't matter: that's speculation. The change that caused the problem is in this patch somewhere. SPC didn't exhibit the behavior, then they did. I know when they started, I know the commits, so there's a diff to read through. You could read through it or you could argue with me or you could leave me alone while I read through it.
> The number of servers running this code can probably be counted on one hand.
If it's already a problem now, then it will be a bigger problem when everyone has whatever change has caused this. Baest was maybe 1% of the total number of POST reqs going to FSE and then suddenly it was 80%, and that figure includes interactive users, it includes every time POST landed on that server. I posted about this. You can dig up the screenshots. Whatever. Like I said, I'm not going to argue with you about it any more.
@p@Terry@colonelj@sjw@feld@iamtakingiteasy@colonelj@lanodan@Moon User-specific inboxes were a mistake. They literally serve zero purpose since the scope/list of recepients is already defined in activity that's being sent out.
@p@Moon@Terry@colonelj@colonelj@iamtakingiteasy@lanodan@mint@sjw I also have another MR that wasn't merged yet that will prevent Pleroma from retrying a delivery if it gets a 4xx back. I have been using that for weeks.
In either case, though, the server should try to deliver it once. The issue with bae.st/SPC/FSE was that if a 150 people on FSE are following someone on one of the other two servers, and that person clicks "Like" on a post, then that's 150 attempts to deliver the Like, and multiply that by the number of times that person clicks Like and the size of their bio. If it's Create instead of Like, it's an even bigger deal.
@p@Moon@Terry@colonelj@sjw@iamtakingiteasy@colonelj@lanodan@mint What the fuck. The first rule of inbox is to return 200 unless you want other servers to retry.
> The first rule of inbox is to return 200 unless you want other servers to retry.
No, you dipshit. Go read RFC2616, which is both very readable and also very clear about when to retry and has not substantially changed. Here is the relevant section: https://www.rfc-editor.org/rfc/rfc2616#section-10.4.10 . 4xx errors should not be retried.
@p@Terry@colonelj@sjw@iamtakingiteasy@colonelj@lanodan@mint@Moon As a person with actual experience building Fediverse software, I would recommend returning a 200 unless you want other servers to retry.
As a person with actual experience having to fix your fuckups when graf asks you why something is broken and you suddenly go idle, I am not interested in your opinion on anything.
Additionally, I don't know how many times you have to hear "I view you as a liability and do not wish to interact with you" but it seems to me like it's counterproductive for you to keep trying to force an interaction. feld may occasionally annoy me, but I respect him. I do not respect you, I do not value your contributions or your opinion, I am never happy to see you. You're in the same category as "Comrade Lecter" as far as I am concerned, and I wish you would fuck off to Nostr.
"I have put you on a permanent ignore, public and private. I have found you disturbing, rude and generally not worth talking to. according to the channels you hang on, it strengthens the effect of wanting to put you on ignore because of my lack of interest in you as a person. This message is not meant to be rude to you, just to inform you that I wont see any of what you type from now on."
@NonPlayableClown@7666@alex Fedi works fine no matter what anyone in this thread does, but I think fedi goes better if the people doing the work eliminate sources of friction. There are people I won't work with, things I won't work with, and it's better to just avoid them, so the less effort spent on this, the better.
The problem with Gleason is that, after he lost his temper at the actual Pleroma devs, he burned them: accused them of complicity in child-trafficking rings, insulted them personally and professionally, declared that Pleroma was dead, etc. Anyone that does that kind of thing is a liability: it's almost too obvious to say that you don't want to work with a guy that has a history of burning people he works with.
But he's got this problem where if you ask him to leave, he gets pissy. So I tell him I don't want to deal with him, he keeps bothering me. He's kicked out of the Pleroma Gitlab and he harasses the devs and demands that they un-ban him. That's another red flag: bartender tells a guy he's been over-served, the guy starts trying to pick a fight with the bartender, he gets bounced, then tries to fight the bouncer, then keeps trying to go back to that bar and keeps demanding to be let in. (Only Gleason is cold-sober, as far as I know.)
Then, later, he starts complaining of persecution. I don't bother Gleason, he just starts replying to me. I don't see the Pleroma devs bothering him, I see him jumping in to bother them. lain takes a vacation and comes back and Gleason replies "Un-ban me from the Pleroma gitlab!" So, someone that behaves this way and claims he's the one that is persecuted, that is a person that is either a lunatic or a low-grade sociopath.
So, overall, I'd rather just not deal with him. My life is better that way.
>The problem with Gleason is that, after he lost his temper at the actual Pleroma devs, he burned them: accused them of complicity in child-trafficking rings, insulted them personally and professionally, declared that Pleroma was dead, etc. Anyone that does that kind of thing is a liability: it's almost too obvious to say that you don't want to work with a guy that has a history of burning people he works with.
Do you have a link, or screenshots to that?
If not did he butt heads with @lain or something?
Plenty, yes. He shat the bed and then posted this: https://web.archive.org/web/20230330163111/https://blog.alexgleason.me/pleroma-is-dead/ . He then claimed that billionaire Leo Radvinsky personally called the shot, his evidence of this being that MFC paid some Pleroma devs for some custom work years ago. (Apparently none of this bothered him until after he got banned.) That was the last straw as far as I was concerned: he appropriated their code and then shat on them.
> If not did he butt heads with lain or something?
lain is too classy to roll around in the mud with Gleason, which is why it was more uncalled for. (lain's clearly wiser than me.)
> I'm willing to hear you out.
It's stuff that's best not dug up, but Gleason starts making these claims. I have been telling him to fuck off for years, and I tell him the same shit every time he arrives, and he still won't fuck off: either he is getting what he wants from the interaction or he is trying to wear me down. I'm too stubborn for the latter, so it's probably the former.
I still feel like I'm missing some context, some links in archive.org are broken that seemed to hold more details in the matter.
As for my own observation is this:
#1 Alex can be a dick.
Eh... I'm dick myself so I can get why people can get rubbed the wrong way.
#2 Transphobic???
I'm scratching my head on this one.
This seems like this more of a political scare then anything else.
Nothing outside of the article is really showing Alex antagonizing trans devs.
Also looking at the article itself: https://web.archive.org/web/20230331173749/https://blog.alexgleason.me/trans/
The long story short is that he doesn't care what one does to themselves but believes that sex biological.
And funny enough it seems to make Alex partially right in his article, assuming that the "hit piece" was partly motivated by some trans devs taking a stance and leaving.
If Alex was constantly telling them constantly to go kys, yeah I can see that as a problem for the dev team. But from what I saw provided, not so much other to you XD.
#3 commits
This feels really silly to get upset about it.
Couldn't the dev team just make separate branches for your own stuff. It's the shinning example of what git is best used for.
This seems like an organizational issue more than anything else.
The team did give Alex the permissions to allow merging onto dev.
If merging to the branch needed more code review or approval I think that should have been more established. Or at the very least make a pilot tier and have different people to approve that merge into that branch.
This is why I feel like I'm missing some context, as stated earlier.
#4 He doesn't like Leo Ranvinsky...
I have no idea who he is so I can't comment on it.
#5 Pedojacketing???
I mean, yeah a lot people on fedi fap to loli, which is pretty gross if you ask me.
As for the "kidschat" and MyFreeCams, I can't find anything online about that relation. So unless Alex got a direct and reliable source about it that's pretty gay.
Also shit man, Alex you got get on that LEGAL porn money. I mean you willing to work with coomers for 1k a month lmfao.
Ok I understand partly why you don't like him, and that's fine.
You can always just make fun of him and then mute the thread when things get boring.
...Or you can be a bitch and defederate for hurt feelings :SeethingDanielS:
jk
Also I do find this screenshot funny, though I will say that it isn't just devs fapping to anime characters, it's also furry shit too.
> Eh... I'm dick myself so I can get why people can get rubbed the wrong way.
Oh, no, I'm terrible. The difference is that I don't spend my time trying to get noticed by people that have repeatedly said they don't want my company. I try to spend my time on things that are worth it.
> Nothing outside of the article is really showing Alex antagonizing trans devs.
Gleason mainly wants to be able to claim persecution. In reality, if he didn't ever bother me, I wouldn't have a reason to notice anything he's doing, and I suspect that this is the case for most people.
> But from what I saw provided, not so much other to you XD.
This is the claim he makes. The first time I talked to him, he claimed that he had been kicked out of veganism for being transphobic, he'd been kicked out of feminism for being transphobic, and now he picks a fight with the Pleroma devs and says he was kicked off the Pleroma server for being transphobic. It looks like when he fucks up, he claims he's being persecuted for transphobia.
> #4 He doesn't like Leo Ranvinsky...
He's got a Wikipedia page, definitely a dude with his fingers in a few less than respectable pies. I don't really care if he doesn't like Radvinsky, but the idea that a billionaire is stalking him across the internet is absurd.
> As for the "kidschat" and MyFreeCams, I can't find anything online about that relation. So unless Alex got a direct and reliable source about it that's pretty gay.
That blog post he linked to had the "evidence", which was essentially that the guy that owned MFC had previously funded some forum software as well, and one of the forums that used it was called "KidsChat". You can find it if you dig through his old posts.
> Ok I understand partly why you don't like him, and that's fine.
It's mainly that if you tell someone to quit bothering you and they just follow you around, that's a really good sign that you were correct to not associate with them. So he starts ranting about enemies, and I'm only his "enemy" to the extent that I tell him to fuck himself every time he bothers me. It's like some homeless schizo thinking you're an enemy because you tell him not to get near you. He's like...did you ever know a girl that had "crazy ex syndrome" but you never even dated her? It's that kind of thing.
> then mute the thread
The thread was about the bug, which actually was important. Gleason wanders in. I don't notice, because I'd muted him anyway, but then people start replying to him. Can't mute a sub-thread. But it's not reasonable anyway: "You can just leave the room every time he enters the room" is the equivalent of letting him decide who I talk to and what I talk about. I don't wish to talk to him about anything, it strikes me as reasonable to say I get to decide who I associate with. I don't want to associate with him, and it apparently irks him, or for some reason related to posturing on the internet he wants to constantly get told to go away. Beyond me to figure it out: there are so many interesting people on here that I feel like I'm wasting fedi if I burn time talking to someone like this guy.
That's why I say this stuff isn't worth bringing up: I don't feel like I got anything done, just shat on Gleason with the hope he'd go away, and he never does. Is that a good use of my time? What makes him think it's a good use of his time to keep trying to talk to people that do not want to talk to him? Maybe he's hoping to provoke some reaction that he can take to Nostr and say "They chased me out of fedi for having the courage to speak my convictions!" Maybe he thinks he's got to avenge any slight, real or imagined. Maybe he wants to stir up fights to appear relevant. (That's certainly not a new tactic for vegans: http://slatestarcodex.com/2014/12/17/the-toxoplasma-of-rage/ . For all of his noise, fewer and fewer people bother with Soapbox: http://demo.fedilist.com/source-code .) Or maybe he's a fed. No telling.
Anyway, I don't wanna further shit up Moon's server with this nonsense. I am a guest here.
This is the claim he makes. The first time I talked to him, he claimed that he had been kicked out of veganism for being transphobic, he'd been kicked out of feminism for being transphobic, and now he picks a fight with the Pleroma devs and says he was kicked off the Pleroma server for being transphobic. It looks like when he fucks up, he claims he's being persecuted for transphobia.
But the article literally says:
To the ones not really aware a large chunk of the pleroma community was transgender people. And of course it doesn't really mixes well with someone having an article named Why I don't support transgender ideology which would trigger a knee-jerk reaction on basically anyone supporting the cause. And while this has worked okay, the disaster happened anyway, we lost a large chunk of the community, we lost collaboration with glitch-social, which killed MastoFE. In retrospect there should have been much more clarity and better stances, for example community guidelines to reverse-signal on accepting someone which is basically anti-trans.
So for this one case, there is some validity to it, unless whoever made that article is just full of shit to begin with.
That blog post he linked to had the "evidence", which was essentially that the guy that owned MFC had previously funded some forum software as well, and one of the forums that used it was called "KidsChat". You can find it if you dig through his old posts.
Ok I was looking more into this after Alex linked up to the article. At first my speculation was that it was google being retarded on placing ads. But after your response that sounds pretty bad. I don't want to confirm this because I don't want to go on a pedo site to see if freecam ads exist on there. But most importantly did anyone bring this up to Leo or admins on freecams? If not that's pretty fucked. Now as long as Leo or Co removed ties and didn't act like :SeethingDanielS: with Undu then cool. If not then that's not good.
The thread was about the bug, which actually was important. Gleason wanders in. I don't notice, because I'd muted him anyway, but then people start replying to him.
Well technically only you replied to him after this comment:
What the fuck. The first rule of inbox is to return 200 unless you want other servers to retry.
I only got involved when things got very spicy. You could have ignored him at that point. Plus Alex isn't being malicious with the statement, though he is being dick 🤣
I'll end off with this. Despite people's dislike of one an other, we still do not want instances going down. Yeah me and bloomfer didn't like Graf, but we still didn't like it when Poast was down. Mostly because that's 10k users (some of which I do like) cut off social media. So we tried help out when we can, of course through middle men because... we were defed 😆
I think its safe to say we all want FSE to be online. I think that we can all agree on that.
Okay, here: I don't want to deal with this further. Here's what I have to say, it's unlisted, I don't want to deal with any more Gleason-anything. Since the bug is fixed and the thread's got no upside, I'll just mute the thread after I post this.
> But the article literally says:
If you're quoting lanodan's piece, that was after the fact. I have never seen lanodan lie about anything, though we don't always have the same opinions about things, lanodan is honest and intelligent and good-natured. So, I was there, too, and for the time before that, and a write-up is about conclusions: I know lanodan's conclusions, but I don't think the blog post has any information that I didn't have before this conversation.
So, years before Gleason shits the bed, he does these terrible things like fuck up how URLs work (so that you cannot use the local search bar to get a remote post, because he thinks in terms of silo'd instances instead of a network, because he is retarded: he could fix it by returning the right JSON or making paths in the backend conform to the paths he wants in the frontend, but he cannot cope with any environment in which a browser has not executed his JavaScript, so he cannot understand S2S), he fucks up the replies, I make the casual remark "Did Soapbox fuck up replies? That's awful." and he enters the thread out of nowhere to start screaming, I address the problem with the code, and then he loses his shit entirely. This is not how you deal with a bug; at that point, I conclude that he was either malicious or so incompetent that it was impossible to distinguish from malice. (You look at his trajectory, vegan feminists, Gab, Truth Social, now Nostr. For all I know he actually is on someone's payroll.) He wants to make it about his beliefs: the reality is that he's a punchy dipshit with short man syndrome and is not good at writing software or architecting systems. I don't give a damn that he's a feminist or that he's "transphobic" any more than I give a damn about someone being a "men's rights activist" or trans. His knee-jerk criticism doesn't work on me. Because I made that clear long before he got banned from git.pleroma.social, he also cannot claim that I've jumped on a bandwagon: here's this post from way before the ban, he's tagged in it.
There are people that don't like him, sure, there are people that don't like anyone. He wants to blame transphobia, but he can't in my case, because I don't care if he's "transphobic". I just care that his code is shit, and the extent to which I care is that I try to keep it from affecting me. That's it.
So, sure, he was unpopular but even at the expense of those things, they kept him around on principle: everyone was professional around the repo. That remained the case until he ignored the release process and started fights with the other devs when that change was undone. You check how many times he mentions lain in that "Pleroma is Dead" piece and lain went to bat for him, and he shat on the hospitality, then harassed lain. lain was the only one with any goodwill towards the guy, and this is the person he keeps attacking. Long before he was kicked off for throwing a tantrum in the Github repo, when I was complaining about his code and speculating that he's a fed, lain's defending him, telling him in public that the reaction to the reply fuckup is unfair. If anyone has a slight chance of being nice to Gleason, it's lain. And that piece he wrote is more vicious towards lain than anyone else. This is because he wants to have the "problem": he's shit-stirring, not because he actually cares about what he claims to care about, but because he wants the reaction. So he wants a fight or he wants to be the topic of discussion or he's just so goddamn short-sighted that he doesn't have a handle on himself.
> I don't want to confirm this because I don't want to go on a pedo site to see if freecam ads exist on there.
Nah, like, it's the equivalent of declaring the EU a Nazi organization because they funded Mastodon development and pieville.net uses Mastodon.
> Well technically only you replied to him after this comment:
> Plus Alex isn't being malicious with the statement,
If he's replying to me, he's shit-stirring. Not the first time he has done the "I'm not touching you." Maybe his problem is that he doesn't have a handle on himself. He has on at least two occasions begged graf to ask me to stop saying mean things to him. He has asked me why we can't be friends and accused me of "bullying" him. and I tell him, unambiguously, that I don't wish to deal with him and won't have a reason to talk to him if he stops talking to me.
He was an ass from the beginning, the bottom line is that I don't plan to associate with him. Since he has a history of shitting on people that he has worked with, and people that were kind to him when no one else would be, and lying about them and lying about why and lying to me about events I was present for, I would not accept his labor for free if it came at the cost of dealing with him directly. He is a liability, he is not someone you want on your team. As his years of effort have not resulted in any movement towards his stated goal, either he is retarded and incompetent, or he's evil: the difference doesn't mean anything to me. Maybe if he figures out how to get his testosterone levels back to normal he'll stop this behavior, and maybe then he'll be worth dealing with, and the world is huge: he can find other people that are willing to deal with him. I am not.
Like I said, I'd like not to shit up SPC with this. I don't even like shitting FSE up with it. Even if it didn't shit up the place, I don't like devoting time to it. That is the reason that I was annoyed to begin with: here's Gleason again, and he could go do something productive or he could keep trying to tickle my balls with his beard. If you want to discuss further, wait until FSE's back up and the dust settles, and even then, ultimately I participated in and witnessed events, I drew conclusions, and unless there's new information, you have my conclusions and you can do whatever you want with them. You have questions about what lanodan wrote, you can ask lanodan: I can only really say what I was there for, and I said what I said.
> I think its safe to say we all want FSE to be online. I think that we can all agree on that.
Ha, you'd be surprised. Anyway, it'll be back, one way or another. It can't be killed and I will ensure that it out-lives me.
@mangeurdenuage@Moon@Terry@colonelj@colonelj@iamtakingiteasy@lanodan@mint@p@sjw there's some some chances that yes. Can Moon check specifically what instances are the culprit, then contact these instances owners to see if there's a particular user loading a shitload of user data abnormally?
Is there anything preventing someone from signing up on a random instance and then using some sort of modified fediverse app to then scrape data?
Also, sorry to bother you further, but is there a way to check what followers of mine, or really anyone else here, that are inactive/have a suspiciously low amount of posts compared to their profile date one way or another? This could help in kicking out scrapers
@coolboymew@Terry@colonelj@colonelj@iamtakingiteasy@lanodan@mangeurdenuage@mint@p@sjw the entity scraping your account was not logged in, they were just getting forbidden errors, but several times a second. Getting a strong impression that the "academics" that are "researching the Fediverse" are total dumbasses.
the ec2 thing was different, I think they have an account here and are scraping.
>Getting a strong impression that the "academics" that are "researching the Fediverse" are total dumbasses.
Honestly, everything I've seen from the "propaganda war" front are usually lead by the biggest brainlets, part of which is because all of this is a massive scam for funding useless projects
@coolboymew@Moon@Terry@colonelj@colonelj@iamtakingiteasy@lanodan@mint@p@sjw
> part of which is because all of this is a massive scam for funding useless projects
Imo it's again an issue of "evil" vs "stupidity" it's impossible to distinguish if we want ot be fair but that doesn't me it shouldn't be stopped either way.
@Lyx@Terry@colonelj@iamtakingiteasy@colonelj@lanodan@mint@Zergling_man@Moon@p the Baest server also handles over 100 million http requests a day so a big spike from federation really isn't going to affect it all that much.
Add comment