Feldon This took a few months bit longer than anticipated, but at long last, EQ2Wire is now hosting a complete Forum Archive of the EQ2 Forums, as well as selected Station Forums.
Producing this archive took me weeks of programming and testing to create a Forum Scraper script in PHP which could scan and download entire posts, threads, forums, and users from SOE’s servers without causing undue burden and without scrambling the data from the French, German, and Japanese forums. The script successfully captured not only posts, but every user’s Avatar and even Signatures. I captured approximately 40% of the archive just prior to SOE Live, and the remainder of the archive in the past few weeks. Actual runtime to capture the EQ2 forums was approximately 3 weeks. Not bad for having no access to the data except externally. For good measure, I also grabbed the Fan Faire and SOE Live forums, as well as the Player Studio and Census (Data Feeds) forums.
Those of you who have extensively linked to old EQ2 forum threads will be happy to note that I preserved the thread IDs so you can simply remap your URLs and they will point to this archive. Despite over 2.3 million posts in 210,000 threads taking up over 4GB of data, the archive is completely searchable by topic, post, or username.
Hopefully EQ2Wire readers will find this Archive useful!
very nice, thanks for doing this
Thanks!!!!!
Wow, I cannot imagine the amount of work that took.
Feldon is amazing. I seriously do not know how he finds the time to gift us all with this site and EQ2U.
I know that Dethdlr works pretty manically on the sites too yet Feldon is the backbone.
You’ve got a display issue with UTF fancy quotes.
http://archive.eq2wire.com/showpost.php?p=5802918&postcount=173
https://www.google.com/search?q=%C3%A2%E2%82%AC%C5%93
Your meta tag says charset=ISO-8859-1, but if you change that to charset=UTF-8, they display correctly 🙂
Done. Now does that break anything else?
Also, uberfuzzy’s example above is someone’s signature, not their post.
UPDATE: I fixed those UTF8 signatures. There were only 14 of them.
The bigger problem is the posts. It looks like in an overabundance of caution, I was stripping <br /> tags. Since the EQ2 forums allowed almost any kind of HTML, some posts used <p> or <div> tags, but others used <br /> and I stripped those out. It’s a real mess. 🙁 I’ve run a query and there are over 17,000 posts over 2,000 characters long with no spacing whatsoever. I may have to scrape the “new” Archive just to fix those posts. 🙁
Blown away..again..thanks for all the hard work !
Very cool, thanks Feldon.
Things like this,are why I donate when I can.
Thank you Feldon.
Nice, Feldon. I’m doing an almost identical thing for my employer, but it’s for the UK Patent Office instead of the EQ2 forums.
Great work Feldon. Better to have two archives than none at all. For the record: I like yours better and will use it instead of SOE’s (which lists all boards as empty until you enter them).
Most likely not. Really only affects those stupid fancy quotes, and some non-english chars. I’ve just dealt with that sort of thing in a past job, and knew to check for it ;P
Also, I do want to give proper kudos for making this effort, AND preserving the url structure/IDs
Ahhh, vBulletin! What a message board should be!
Why is it, you can do this, in your “spare” time, without expectations of compensation, yet even the new EQ2 Forums are crud (they REALLY need to break up the class section BY class and not archetype . . . ).
I’d say they should hire you . . . but I’d be afraid they’d corrupt you and you’d put out the same “FINE” work they’ve been doing for years . . .
Corporations: Killing productivity since the invention of writing . . . (so quite a while now).
vBulletin 3.x was great. vBulletin 4.x and vBulletin 5 are complete rubbish. InternetBrands bought out vBulletin and ran off all the good developers and then sued them for daring to write XenForo. Fortunately that lawsuit was finally settled last month and XenForo were victorious.
Unfortunately the implementation by SOE of XenForo turned off a bunch of cool features and adopted a very restrictive and visually distracting style. It’s sad as someone who has administrated forums for 12 years to see some of the policies and decisions being made. XenForo is still in its infancy, but it’s not nearly as crippled as it’s been implemented here.
I’d be in meetings all day and we’d still be trying to get EQ2U launched. 🙁
Outstanding, Feldon. Your work here is nothing short of brilliant. We are so fortunate to have you and your talents.
Thank you.
Congrats!
French board is missing all special characters aka: éèàçùêô for the most used
And that would be the consequence of setting the charset globally to UTF8. 🙁
The characters are correct on the French boards if you force Latin rendering.
I will see if I can fix the UTF8 quotes mentioned by uberfuzzy another way.
mmmh, after more checking, it only shows “?” in the main french board page and not in the actual threads.
Looks like most threads are fine – but I was looking at this one earlier, and it hurts my eyes…
http://archive.eq2wire.com/showthread.php?t=516455
Some of the update notes turned out pretty bad for some reason. I can fix them as I’m made aware of them.
Compared to what it should look like
http://forums.station.sony.com/eq2/index.php?threads/birth-to-badassery-a-conjurors-guide.526455/
My point exactly, Feldon . . . my point exactly. Thank you for all that you do.
Bureaucracy: The mortal fear that someone, somewhere, is working efficiently and without our permission.
nice work sir!
Feldon would get hired, pull out all his hair from the frustration, and then be transferred to EQNext or worse, and we would never see him again.
Glad you are sticking to your independent spirit, Feldon!
The only thing worse then doing UTF8 on the web is time/date math (re:april 48th). Sorry for opening my mouth.