+- Drunkard's Walk Forums (http://www.accessdenied-rms.net/forums)
+-- Forum: General (http://www.accessdenied-rms.net/forums/forumdisplay.php?fid=1)
+--- Forum: Forums (http://www.accessdenied-rms.net/forums/forumdisplay.php?fid=3)
+--- Thread: 500 luftballoons (/showthread.php?tid=3534)
- Black Aeronaut - 06-28-2017
Yeesh. Talk about bad timing.
And I just started my new job with will have me working 12-hour nights six days a week, so I won't be able to do much research...
- Inquisitive Raven - 06-29-2017
Yowza! Have you managed to back up the forums, yet?
Also, consider this my permission to copy and move anything I've posted here. Not that any of it's all that valuable....
- Bob Schroeck - 06-29-2017
No, nothing's been done at all in the weeks since this was last discussed.
-- Bob
---------
Then the horns kicked in...
...and my shoes began to squeak.
- Ankhani - 06-29-2017
I can have a spidered copy of the forums in about 48-60 hours. I don't know how it'll impact pageviews. Shall I press the button?
---
The Master said: "It is all in vain! I have never yet seen a man who can perceive his own faults and bring the charge home against himself."
>Analects: Book V, Chaper XXVI
- Bob Schroeck - 06-29-2017
Do it. The Terms of Service specifically say we have both a right and an obligation to back up the forums, and I'm not willing to got through another platform change without an archived copy of the content.
As for pageviews, well, we have over 145000 currently left in the contribution bank, and while I don't think they're likely to get exhausted I can refresh that at any time.
I'd suggest filtering out the threads that got blanked in the crash ten years or so ago to save on pageviews, but honestly, you'd need to view the page anyway to know the thread's been blanked, so don't worry about it. If we have to restore the content here or elsewhere, we'll filter them out then.
Thank you, Akhani. That's one less worry off my mind.
-- Bob
---------
Then the horns kicked in...
...and my shoes began to squeak.
- Bob Schroeck - 06-29-2017
Oh, and a thought on detecting spiders/bots. Folks might remember that a couple years ago, we had an incident where something happened -- an out-of-control bot or something -- and we had a few hundred thousand pageviews eaten up in a day or three.
Yuku never noticed. I had to bring it to their attention and then point out strenuously that it was a freaking obvious anomaly in our usage patterns before they'd credit us back the pageviews.
I haven't seen any evidence that their system has changed in the ensuing years. And given how many boards they host, I suspect that it would be beyond their capabilities to monitor each and every board for unusual usage patterns; doing so would slow down things so far that everyday use would be impossible. And how would they differentiate between "unusual usage" and a board with a sudden gain in popularity?
And honestly, any kind of usage is money in their pockets, whether it's from ads or pageview contributions. It's not in their financial interest to analyze usage and block "improper" access -- it loses them money on both ends of the transaction. If they're monitoring traffic in any way, It's probably some DHS- or NSA-mandated content/keyword tracking.
-- Bob
---------
Then the horns kicked in...
...and my shoes began to squeak.
- Ankhani - 06-29-2017
Backup is running; I'll let you know when it finishes. This is a pretty big site, though, so I expect it will take some time. I don't think this will be a problem, but it is worth it to note, I can't get any private boards with this. If there are any that are hidden behind a login, you'll have to backup those some other way.
---
The Master said: "It is all in vain! I have never yet seen a man who can perceive his own faults and bring the charge home against himself."
>Analects: Book V, Chaper XXVI
- Bob Schroeck - 06-30-2017
I'm waiting back to hear from Josh about hosting a forum at warriors-alpha.net, but I'm also looking into hosting it at Access Denied. It depends on how big the scrape turns out to be, and how much space the board software takes up.
I've looked at a few free board packages already, and so far I'm liking MyBB the best of those I've seen. If anyone has any suggestions or preferences, I'm open to them.
-- Bob
---------
Then the horns kicked in...
...and my shoes began to squeak.
- robkelk - 06-30-2017
Bob Schroeck Wrote:...What do we want to be able to do?
I've looked at a few free board packages already, and so far I'm liking MyBB the best of those I've seen. If anyone has any suggestions or preferences, I'm open to them.
http://en.wikipedia.org/wiki/Compariso ... m_software
Right now, we have a flat forum with partial user-selectable themes, image attachments, and unread message tracking.
--
Rob Kelk
"Governments have no right to question the loyalty of those who oppose
them. Adversaries remain citizens of the same state, common subjects of
the same sovereign, servants of the same law."
- Michael Ignatieff, addressing Stanford University in 2012
- Bob Schroeck - 06-30-2017
I've visited that page, too -- it was one of my longer stops this morning.
I don't want to lose any features we already have, like requiring user registration to cut down on the spammers, supporting video embedding...
Maybe it's time for a wish list of forum upgrades, and we pick a package that gives us as many of those as possible.
-- Bob
---------
Then the horns kicked in...
...and my shoes began to squeak.
- LynnInDenver - 07-01-2017
One of my recommendations for a feature? The ability to delete one's account. This can either be a permanent deactivation, or an outright deletion, but the idea is that, if needed, one can at least remove oneself from the member rolls, even if they can't necessarily delete their contributions to discussions.
It's something I've kind of tripped over again and again, and eternally frustrated by, in terms of having left forums, but unable to do anything about my account being there.
--
"You know how parents tell you everything's going to fine, but you know they're lying to make you feel better? Everything's going to be fine." - The Doctor
- Inquisitive Raven - 07-01-2017
It occurs to me that you might not want to move the content in this forum or the Website forum. Both are primarily about issues that can be resolved, hopefully fairly quickly, and anything currently in the Forums forum will be pretty much irrelevant post move, while anything currently in the Website forum would be rendered irrelevant by a major revamp, which I think you were planning if even if you haven't implemented it yet.
Is there any reason to keep anything over a month old in the Marketplace forum? Okay, I don't know if it's even possible to just move the few most recent threads from a forum.
- Labster - 07-01-2017
From a technical point of view, it's a complete pain in the ass to delete accounts. If you keep the text content they contributed to the discussion, it's still going to need to have a writer in the users table. So you either replace their account with a dummy account for every post -- which helps no one. Or you keep it linked to the deleted account, and then anyone can come on later and impersonate you by creating a new account. Really, disabling the account (and hopefully deleting non-public data) is the best way to go.
Maybe the Eyrie would be willing to host us on it's glorious DCForum 6.23. What, hey, it went to 6.25? There was apparently an upgrade in 2003. Let me download that.
/google-fu for 10 minutes
/cast arcane archive.org incantation
/read eldritch perlonomicon
Did you know that you can use a single quote character to delimit namespaces in Perl? I did because I'm 1337 like that, but BBEdit's syntax highlighter sure doesn't know. Anyway this appears to predate CGI.pm which is um, impressive!
-- ∇×V
- robkelk - 07-01-2017
Inquisitive Raven Wrote:It occurs to me that you might not want to move the content in this forum or the Website forum. ...We don't want to move any content. However,
Bob Schroeck Wrote:I just got an email -- the forums are being migrated to Tapatalk in about a week, whether I want it or not.so all of our content is moving whether we want it to move or not. At the moment, we're discussing where we want it to move.
Inquisitive Raven Wrote:Is there any reason to keep anything over a month old in the Marketplace forum? Okay, I don't know if it's even possible to just move the few most recent threads from a forum.Deleting old threads (or new threads) is usually a trivial process for an admin, no matter which software is running in the background.
--
Rob Kelk
"Governments have no right to question the loyalty of those who oppose
them. Adversaries remain citizens of the same state, common subjects of
the same sovereign, servants of the same law."
- Michael Ignatieff, addressing Stanford University in 2012
- LynnInDenver - 07-01-2017
Oh, Bob, there is another thing to consider; board policies. If anything, policies against the sorts of things that places like ATT has struggled with lately. I actually ran into an acknowledged sockpuppet over on BoardGameGeek (I'm just a lurker over there) who was trouncing all over a thread about "this was mildly morally ambiguous and should have been considered BLACK MORALLY WRONG by the OP" and refused to see that he was fighting a losing battle, particularly since they were doing it anonymously, having acknowledged they had another account they didn't want the argument connected to. It's that last part that actually really raised my hackles up; I have done the "acknowledged sockpuppet" myself, but I was very much making an above board effort to be open and obvious about which account was considered my "primary" at the time.
I also know some boards have features for "early user posting restrictions" (10-20 posts and replies require moderation, no links allowed, no images) and "verification" (Pinside does very limited rights in certain areas like Marketplace until you donate and/or get vouched for by a verified member). It might be quite useful to have the board support those sorts of additions, even if you don't implement them immediately.
--
"You know how parents tell you everything's going to fine, but you know they're lying to make you feel better? Everything's going to be fine." - The Doctor
- robkelk - 07-01-2017
IMPORTANT NOTE FOR EVERYONE
Your forum avatar, any images you've uploaded here, your signature file, and anything else associated with your personal account is not saved at drunkardswalkforums.yuku.com - it's saved at u.yuku.com . A forum scrape might not capture it.
Save a local copy of anything you've uploaded here.
--
Rob Kelk
"Governments have no right to question the loyalty of those who oppose
them. Adversaries remain citizens of the same state, common subjects of
the same sovereign, servants of the same law."
- Michael Ignatieff, addressing Stanford University in 2012
- Ankhani - 07-01-2017
The backup has finished, but not without errors. As I have never tried to mirror a site as large as this my bot tripped what was supposed to be only a code sanity check of "Quit on 100000 links crawled". Given the top-down method of the bot, this means that it only got down deep enough to save the first and last few pages of threads. This is is only a problem for the big boards with more than about 100 topics (when you get more than 6 pages of threads).
I'm going to try and crawl those boards individually and see if I can get a little deeper in this time. I'll start with one of the smaller ones to see if this gets any better results, and then I'll be back on things.
---
The Master said: "It is all in vain! I have never yet seen a man who can perceive his own faults and bring the charge home against himself."
>Analects: Book V, Chaper XXVI
- Ankhani - 07-02-2017
Alright, large forums have been backed up individually without (apparent) issue. I am reasonably confident I have local copies of all availible forum posts, as of at least thursday.
Format, if you want to use it, is around 3.5GB of HTML files with copies of all embedded images all internal links rewritten to remain within the local files (external links still point out to the web).
---
The Master said: "It is all in vain! I have never yet seen a man who can perceive his own faults and bring the charge home against himself."
>Analects: Book V, Chaper XXVI
- Black Aeronaut - 07-02-2017
Brilliant work, Ankhani! Even if we can't import the threads into a new message board, at least we can maintain an archive.
- Bob Schroeck - 07-02-2017
Thank you, Ankhani! Next comes the choice of board software and where we host it... and once's that's determined, we can take a shot at importing the archive.
-- Bob
---------
Then the horns kicked in...
...and my shoes began to squeak.
- Black Aeronaut - 07-02-2017
Beehive looks nice. According to Wikipedia, it's one of the more secure platforms. Best of all, it has a lot of the features we like, plus it can import and export MySQL database tables. Which means if we can convert our scrapes of the forums into a database, then we're golden.
Alternatively, there's XenForo, which if people here are familiar with if they go to SpaceBattles or Sufficient Velocity on a regular basis. It's some very nice software, but unfortunately it's not free. It's $140 for the basic license, and then another $40 every year afterwards to maintain access to XenForo's tech support. Other features cost extra, but to my knowledge the basic license should suffice for our needs. Though I recommend someone here contact Xon over at Sufficient Velocity as he's been contributing significantly to their user-generated add-ons.
- Bob Schroeck - 07-03-2017
Quote:if we can convert our scrapes of the forums into a database, then we're golden.We should be able to automate a fair amount of it, he said optimistically.
Beehive does look promising, although I'd ask folks to check out their live demo forum and let me know if you're comfortable with the interface. It may be "I like what I'm used to" talking, but I kinda like our "top level board listing" format. (Then again, that may be one of the things that is customizable; the demo is only one board and doesn't show what the multiboard interface looks like.)
Its goal of efficiency in space, bandwidth, etc. is definitely a big plus for me. I have to go back to my ISP and find out what my storage limits currently are -- all I remember is that my website is comfortably under them -- and how much it'll cost to expand them if it's necessary to host 3.5 gigs of historical data (yes, I know it'll be smaller when we extract the content) + room for the future. I'm going to have to actually talk to someone there, because their website has changed since the last time I went digging for info there, and now I can't find the relevant page(s). I'm pretty sure they already have MySQL available.
Might be a week and a half at least until I can do this -- what with the holiday, our upcoming vacation and all.
-- Bob
---------
Then the horns kicked in...
...and my shoes began to squeak.
- LynnInDenver - 07-03-2017
As long as it's not "flat", and doesn't come with the "use Tapatalk for a mobile experience" micro ad, I'm golden.
--
"You know how parents tell you everything's going to fine, but you know they're lying to make you feel better? Everything's going to be fine." - The Doctor
- robkelk - 07-03-2017
Bob Schroeck Wrote:I suspect that, if we want a user interface similar to the ezBoard/Yuku interface, we might have to go with vBulletin. That's a $249 outlay for the software. www.vbulletin.com/en/features/Quote:if we can convert our scrapes of the forums into a database, then we're golden.We should be able to automate a fair amount of it, he said optimistically.
Beehive does look promising, although I'd ask folks to check out their live demo forum and let me know if you're comfortable with the interface. It may be "I like what I'm used to" talking, but I kinda like our "top level board listing" format. (Then again, that may be one of the things that is customizable; the demo is only one board and doesn't show what the multiboard interface looks like.)
...
(Yes, I am suggesting vBulletin because I already know the interface. Example vBulletin forum: http://forums.sjgames.com/index.php )
--
Rob Kelk
"Governments have no right to question the loyalty of those who oppose
them. Adversaries remain citizens of the same state, common subjects of
the same sovereign, servants of the same law."
- Michael Ignatieff, addressing Stanford University in 2012
- Black Aeronaut - 07-03-2017
Beehive's interface is a little unorthodox.... It more reminds me of the threaded forums you might see on CraigsList.
I'd have to see a live implementation to get a better idea.