SimplePress Forum Import Template

[netweb]

Introduce Forum Importer template for use with importing from Simple Press 5.1.4

[johnjamesjacoby]

Nice.

[MacItaly]

Hi Stephen,
this is the report of the first attempt to use this importer.
Hope that it helps.
Thanks

Mac

First test:

ENVIRONMENT:
Simple Press 5.1.4 with 2 Groups, 21 forums, 2,468 topics, 57,996 posts, 556 users
BBpress 2.2.2
WP 3.4.2 users 556
both SP and BBPress in the same db

MAMP localhost su Mac OS
PHP 5.3.5 - MYSQL 5.5.9 - COLLATION utf8_general_ci

================== START TEST 1 =======================

Added Simplepress5.php in converter folder, choosen Simplepress5 from dropdown menu in import forum and started conversion without import users (because same site users).

Row limit= 1,000, delay time= 2, convert users= unchecked, Start Over= unchecked, Purge Previous Import= unchecked

This is the result of yellow import box, at bottom page:

"
Repair any missing information: Continue

WordPress database error: [Unknown column 'sfposts.post_subject' in 'field list']
SELECT convert(sfposts.post_id USING "utf8") AS post_id,convert(sftopics.topic_id USING "utf8") AS topic_id,convert(sfposts.forum_id USING "utf8") AS forum_id,convert(sfposts.topic_id USING "utf8") AS topic_id,convert(sfposts.poster_ip USING "utf8") AS poster_ip,convert(sfposts.user_id USING "utf8") AS user_id,convert(sfposts.post_subject USING "utf8") AS post_subject,convert(sfposts.post_content USING "utf8") AS post_content,convert(sfposts.post_date USING "utf8") AS post_date FROM sim_sfposts AS sfposts LEFT JOIN sim_sftopics AS sftopics USING (topic_id) WHERE sfposts.post_id != sftopics.post_id_held LIMIT 0, 1000

No replies to convert

No tags to convert

Converting topics (2000 - 2999)

Converting topics (1000 - 1999)

Converting topics (0 - 999)

Calculating forum hierarchy (0 - 999)

Converting forums (0 - 999)

Starting Conversion

"

Result of import in db and BBPress menu:

Groups imported: 0 (may be corrispondent to BBPress categories?)

Forum importeds: 21 - number correct - but all Topics = 0 all Replies = 0 all creator = Mac (admin user) all created date= 29 November 2012 time 01:52 fresness= No topics

Topics importeds: 2,468 - number correct - correctly assigned to right forum - but apart the firsts 24 topics (with fresh posts inside) all the rest have date = 1 January 1970 and mostly part of topics erroneously assigned to user "Mac"

Replies imported: 0

======================= END TEST 1 =========================

SECOND TEST

=================== START TEST 2 ======================

refreshed originay DB and restarted process, importing users:
Row limit= 1,000, delay time= 1, convert users= checked, Start Over= unchecked, Purge Previous Import= unchecked

This is the result of yellow import box, at bottom page:

"
Repair any missing information: Continue

WordPress database error: [Unknown column 'sfposts.post_subject' in 'field list']
SELECT convert(sfposts.post_id USING "utf8") AS post_id,convert(sftopics.topic_id USING "utf8") AS topic_id,convert(sfposts.forum_id USING "utf8") AS forum_id,convert(sfposts.topic_id USING "utf8") AS topic_id,convert(sfposts.poster_ip USING "utf8") AS poster_ip,convert(sfposts.user_id USING "utf8") AS user_id,convert(sfposts.post_subject USING "utf8") AS post_subject,convert(sfposts.post_content USING "utf8") AS post_content,convert(sfposts.post_date USING "utf8") AS post_date FROM sim_sfposts AS sfposts LEFT JOIN sim_sftopics AS sftopics USING (topic_id) WHERE sfposts.post_id != sftopics.post_id_held LIMIT 0, 1000

No replies to convert

No tags to convert

Converting topics (2000 - 2999)

Converting topics (1000 - 1999)

Converting topics (0 - 999)

Calculating forum hierarchy (0 - 999)

Converting forums (0 - 999)

Delete users wordpress default passwords (0 - 999)

Converting users (1000 - 1999)

Converting users (0 - 999)

Starting Conversion
"

Results of Forum, Topics, replies, identical to Test 1

But doubled number of users, with all users duplicated adding "imported_"+ username as username

==================== END TEST 2 ========================

[netweb]

@MacItaly,

Many thanks for the detailed report.

Surprisingly I had originally gone down the track that people would be importing everything into bbPress but in the case of Simple:Press because it shares the WordPress _users & _usermeta table the ideal process here is to use the existing WP install, install bbPress and run the import tool.

I have updated SimplePress5.php​ and if you could give it another test that would be grand.

Fixed: User Mapping, Post Dates, Topic 'first post' content, Replies mapping and reply order.

Groups imported: 0 (may be corrispondent to BBPress categories?)

This is now fixed though Simple:Press uses a separate table for 'Groups' sfgroups and at this stage are not being imported as a top level forum category, all subsequent forum hierarchy of forum/sub-forum is supported.

eg. Existing Simple:Press forum layout

• Group Heading
  • Forum One
    • Sub forum one
    • Sub Forum two
  • Forum Two
    • Another sub forum

eg. After import to bbPress

• Forum One
  • Sub forum one
  • Sub Forum two
• Forum Two
  • Another sub forum

Workaround: Most forums won't have to many 'groups' and this shouldn't be a huge issue and after importing create any top level forum categories needed eg. from above 'Group Heading' and assign 'Forum One' & 'Forum Two' to the parent/s.

[MacItaly]

Performing test...
Some lights at the end of the tunnel but still need finetunes (Forum and Topics same results as previous tests).
Importing replies in progress, 57k posts, require time...
Asap I'll come back with report

[MacItaly]

This time seems all good, after rebuilt indexes.

Third test:

ENVIRONMENT
Simple Press 5.1.4 with 2 Groups, 21 forums, 2,468 topics, 57,996 posts, 556 users
BBpress 2.2.3
WP 3.4.2 users 556
both SP and BBPress in the same db

MAMP localhost on Mac OS
PHP 5.3.5 - MYSQL 5.5.9 - COLLATION utf8_general_ci

========= START TEST 3 =================================

Added Simplepress5.php version 2 in converter folder, choosen Simplepress5 from dropdown menu in import forum and started conversion without import users (because same site users) .
Row limit= 1,000, delay time= 1, convert users= unchecked, Start Over= unchecked, Purge Previous Import= unchecked

Result of import in db and BBPress menu:

Groups imported: 0 (not a big deal to reassign manually forum to categories)

Forum importeds: 21 - number correct - but all Topics = 0 all Replies = 0 all creator = Mac (admin user) all created date= 13 December 2012 time 02:16 fresness= No topics

Topics importeds: 2,468 - number correct - correctly assigned to right forum - Replies= 0 Voices= 0 and, as previous tests, apart the last 24 topics (with fresh posts inside) all the rest have date = 1 January 1970 and mostly part of topics assigned to user "Mac"

Replies imported: 55,528 - number incorrect - correctly assigned to right forum (as I could see so far) but under the Forum name, in topic coloumn, there are all red string "(Mismatch)" (corrected after index rebuilding) - Topic, Author and Created data seems corrects (I could not test all)

TABLES:

wp_posts passed from 1,766 records to 60,071
wp_postmeta passed from 3,005 to 237,245 adding 4 records per reply

Importing finished

Performed rebuilding Indexes, one by one

All seems works fine! :) GREAT! (but need to double check, also seems missed 2,468 posts? pair as topics number...)

wp_postmeta passed to 249,797 records

NOTE: not "translated" smile shortcodes and image attachments

=================== END TEST 3 ===========================

Meanwhile yesterday Simple:Press 5.2 seen the light and WP 3.5 too, so I'll upgrade soon and test this new combination.

[netweb]

Thanks heaps for this.

Yes, the counts look correct as your 57,996 posts = 2,468 Topics + 55,528 Replies = 57,996 Total

You will always have "Mismatch" and incorrect counts until the conversion is completed and you have run the "Repair Tool" to update all the counts correctly.

The 'similey' shortcodes I hope to have sorted soon and am looking at this right now as we speak for bbPress & phpBb importers and fingers crossed this shouldn't be to far away.

Image attachments importing is an area I am yet to dive into for any platform as this introduces all kinds of complexities that for now will not be supported by the importers.

If you find anything else that doesn't match up please let me know.

Thanks again,

Stephen

[MacItaly]

Hi Stephen,
about smilies and attachments, I know that many forums like PHPBB3 or SMF and others, codifies attachments with hashes and it's really difficult to find the right file and correctly attach to BBPress replyes but, in Simple Press case this is a little easier because it stores attachments in folders and puts link and the same behaviour is for smiles.
I show you what I mean, the following are HTML codes for smilies:

<img src="/wp-content/sp-resources/forum-smileys/sf-laugh.gif" alt="laugh" width="15" />
<img src="/wp-content/sp-resources/forum-smileys/sf-wink.gif" alt="wink" width="15" />
<img class="bbcode_smiley" title=":)" src="/wp-includes/images/smilies/icon_smile.gif" alt=":)" width="" height="" />

or, about attachments:

<img src="/wp-content/sp-resources/forum-image-uploads/otto/2012/10/screen-4.jpg" alt="screen-4.jpg" width="640" height="960" />

as they results after imported.
Hope that this can help.
Of course it's possible to transfer images and hardcode paths in DB but if the importer takes care of this... :)
Thanks a lot for your great work.

Marco

[netweb]

At the moment I am using converting smilies to use the ones included with WordPress in /wp-includes/images/smilies/ and if WordPress does not have an equivalent it will just render as per the original message eg. /P or :dogcar: etc.

Based on the path for Simple Press you show above they could easily be translated to use the same images and path though I think most people would and I would recommend that after a successful migration there is no need for the Simple Press plugin to remain installed just to keep legacy smilies.

Attachments would be a similar proposition as above and we just need to move them from /sp-resources/ to the WP media library and change the links... I need to think more on the best way forward for attachment imports from all the importers so again I will leave this part alone for now.

[netweb]

Introduce custom callback for converters:

• Can now use to html_entity_decode with preg_replace any custom HTML from existing topics and replies.
• Variables can now be parsed for parser.php adding custom options to the parser as the import is processed.

[netweb]

Revised function naming... in SimplePress5.3.php

[netweb]

SimplePress' default 'Smilies' are now mapped during import to the smilies included with WordPress in the
/core/wp-includes/images/smilies/ folder, these are not hardcoded paths and are enabled/disabled via WordPress Admin -> Settings -> Writing (/wp-admin/options-writing.php) option Convert emoticons like :-) and :-P to graphics on display

@MacItaly It seems that the plugin for attachments is only available as a paid subscriber so I won't be going any further in regard to attachments. Ideally after each migration you would copy all your attachments into the WordPress media library and then edit each post (manually in WP or phpMyAdmin) to relink everything.

[MacItaly]

Hi Stephen,
I'll try the new converter within the weekend.
I understand and agree with your point of view regarding attachments, in effect attachments function is available as paid subscription, even if I believe that by now who uses Simple:Press subscribes also for assistance and premium add ons.
Thanks for your work :)

Mac

[netweb]

Added Simple:Press Locked Topic -> bbPress Open/Closed Topic mapping

[johnjamesjacoby]

(In [4639]) Introduce SimplePress 5 importer. Props netweb. Fixes #2074.